Application Development and Data Modeling on Amazon DynamoDB

  • 670 views
Uploaded on

2014/06/26 …

2014/06/26
AWSプロダクトシリーズ | よくわかるAmazon DynamoDB
David Yanacekによる講演資料

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
670
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
17
Comments
0
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Application Development and Data Modeling on Amazon DynamoDB David Yanacek, Sr. Software Development Engineer, Amazon DynamoDB
  • 2. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Compute Networking & CDN Databases Analytics Deployment & Management Application Services Storage
  • 3. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
  • 4. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture High performance databases throughput elasticity availability latency query flexibility storage capacity operational complexity
  • 5. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Scale Performance Predictable, consistent Degraded performance with scale
  • 6. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture = more problems Data sharding Data caching Provisioning Cluster management Fault management
  • 7. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Undifferentiated heavy lifting
  • 8. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Amazon DynamoDB
  • 9. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Highlights Flexible Durable storage Predictable performance Zero admin Low latency Large scale Seamless scaling Excellent availability
  • 10. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Plan • What is DynamoDB? • In-depth application modeling – Website Session Data (basic data model) – Tic Tac Toe (conditional writes) – Image Tagging (indexes and querying) – Gaming Leaderboard (materialized views) • High-level use cases – Application architecture, data models
  • 11. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Website Session Data basic data model and APIs Web Sessions
  • 12. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Website Session Data Web Sessions
  • 13. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Website Session Data EC2 Instance (web server) Users (first hit to website) set-cookie: { “SessionId”: “af872e”, “History”: [“index.html”] } Web Sessions
  • 14. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Website Session Data: ElastiCache Elastic Load Balancing EC2 Instances (web server) Users ElastiCache • ElastiCache distributed cache • Fast, scales • Node failures cause loss of session data • Adding and Removing nodes is tricky Web Sessions
  • 15. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Website Session Data: RDBMS Elastic Load Balancing Users RDS • Durable • Adding and Removing nodes is tricky • Database blips impact site availability • Traffic can interfere with other DB workloads Web Sessions EC2 Instances (web server)
  • 16. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Website Session Data: DynamoDB Elastic Load Balancing Users • Durable • Transparently scalable • Low latency • High availability • Read consistency • Simple to develop • Simple to operate DynamoDB Web Sessions EC2 Instances (web server)
  • 17. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture DynamoDB Session Handling Sessions Table SessionId Last Accessed Shopping Cart dc32f 2014-06-10T12:... {“Products”: [“Harry Potter DVD”]} 2d231 2014-06-10T12:... {“Products”: [“Amazon Fire Phone”]} e0da3 2014-06-10T12:... {“Products”: [“Kindle Fire”]} 13f47 2014-06-10T12:... Web Sessions
  • 18. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture GetItem: Getting the current session Web Sessions
  • 19. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture PutItem: Updating the session state Web Sessions
  • 20. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Using higher-level SDKs Web Sessions
  • 21. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture DynamoDB Session Handlers (even simpler) • Open-source session plugins in: – Java (Tomcat) – Ruby (Rails) – .NET – PHP – Python (Django) – Node.js (Connect) – … Web Sessions
  • 22. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Offline development • DynamoDB Local: Offline DynamoDB for testing, development Web Sessions
  • 23. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Recap: Website Session Data • DynamoDB’s scalability, performance, availability, and durability, and consistency make it a great fit for website session data • Very easy to use for your website – already implemented open source • Other key/value lookup use cases are natural fits too Web Sessions
  • 24. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Game State conditional writes Social Gaming
  • 25. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe Social Gaming
  • 26. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe { Id : abecd, Players : [ Alice, Bob ], State : STARTED, Turn : Bob, Top-Right : O } Game Item Social Gaming
  • 27. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe Amazon DynamoDB Alice Bob Social Gaming
  • 28. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe Amazon DynamoDB Alice Bob Update: Top-Right : O Turn : Bob Social Gaming
  • 29. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe Amazon DynamoDB Alice Bob Update: Top-Left : X Turn : Alice Social Gaming
  • 30. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Gaming the System Alice Bob (1) Amazon DynamoDB Bob (2) Bob (3) Social Gaming
  • 31. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Gaming the System Alice Bob (1) Amazon DynamoDB Bob (2) Bob (3) Social Gaming
  • 32. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Gaming the System Alice Bob (1) Amazon DynamoDB Bob (2) Bob (3) Social Gaming
  • 33. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Gaming the System Bob (1) Amazon DynamoDB Bob (2) Bob (3) State : STARTED, Turn : Bob, Top-Right : O Social Gaming
  • 34. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Gaming the System Bob (1) Amazon DynamoDB Bob (2) Bob (3) Update: Turn : Alice Top-Left : X Update: Turn : Alice Mid : X State : STARTED, Turn : Bob, Top-Right : O Update: Turn : Alice Low-Right : X Social Gaming
  • 35. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Gaming the System Bob (1) Amazon DynamoDB Bob (2) Bob (3) Update: Turn : Alice Top-Left : X Update: Turn : Alice Mid : X State : STARTED, Turn : Alice, Top-Right : O, Top-Left : X, Mid: X, Low-Right: X Update: Turn : Alice Low-Right : X Social Gaming
  • 36. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Conditional Writes • Accept a write only if values are as expected • Otherwise reject the write • Performance like normal writes Social Gaming
  • 37. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe (Fixed) Bob (1) Amazon DynamoDB Bob (2) Bob (3) Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null State : STARTED, Turn : Bob, Top-Right : O Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null Social Gaming
  • 38. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe (Fixed) Bob (1) Amazon DynamoDB Bob (2) Bob (3) State : STARTED, Turn : Bob, Top-Right : O Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Social Gaming
  • 39. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe (Fixed) Bob (1) Amazon DynamoDB Bob (2) Bob (3) State : STARTED, Turn : Alice, Top-Right : O, Top-Left : X Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null Social Gaming
  • 40. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Recap: Game State • UpdateItem can update specific attributes • PutItem replaces the entire item • Conditional writes synchronize state transitions Social Gaming
  • 41. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging querying tables, secondary indexes Tagging
  • 42. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging: Requirements • Query a user’s images • Query a user’s images by date • Tag other users in images • Query images a user is tagged in Tagging
  • 43. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table Tagging
  • 44. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… Tagging
  • 45. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… Hash and Range primary key schema (composite key) Range: Index is sorted by range key, can do range queries Hash: Equality queries only Tagging
  • 46. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… Bob Query for Bob’s Images Query FROM Images WHERE User=Bob Tagging
  • 47. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging • Query a user’s images • Query a user’s images by date • Tag other users in images • Query images a user is tagged in Tagging
  • 48. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Local Secondary Indexes • Alternate Range Key for your table • More flexible Query patterns • Local to the Hash Key – (more query options within “Bob’s” images) Tagging
  • 49. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… User Date Image Bob 2013-09-05 cf2e2 Bob 2013-10-01 aed4c Bob 2013-10-08 f93bae Alice 2013-09-12 ca61a Table ByDate Local Secondary Index Local Secondary Index on Date Tagging
  • 50. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… User Date Image Bob 2013-09-05 cf2e2 Bob 2013-10-01 aed4c Bob 2013-10-08 f93bae Alice 2013-09-12 ca61a Table ByDate Local Secondary Index Query for Bob’s two most recent imagesQuery FROM Image-Index WHERE User=Bob, DESC, LIMIT 2 Tagging
  • 51. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging • Query a user’s images • Query a user’s images by date • Tag other users in images • Query images a user is tagged in Tagging
  • 52. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table ImageTags Table Tagging
  • 53. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging ImageTags Table Image User aed4c Alice aed4c Bob f93bae Bob Tagging
  • 54. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging ImageTags Table Image User aed4c Alice aed4c Bob f93bae Bob Hash and Range Primary Key Schema Tagging
  • 55. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image User aed4c Alice aed4c Bob f93bae Bob Image Tagging ImageTags Table Bob Query FROM ImageTags WHERE Image=aed4c Tagging
  • 56. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging ImageTags Table Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Bob Tag Alice in Image f93bae Tagging
  • 57. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging • Query a user’s images • Query a user’s images by date • Tag other users in images • Query images a user is tagged in Tagging
  • 58. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Global Secondary Indexes • Alternate Hash and/or Range Key for your table • Even more flexible Query patterns Tagging
  • 59. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging ImageTags Table Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table Tagging
  • 60. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging ImageTags Table Global Secondary Index on User, Image User Image Bob aed4c Bob f93bae Alice aed4c Alice f93bae ByUser Global Secondary Index Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table Alternate Hash and Range Keys Tagging
  • 61. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging ImageTags Table Alice Query for images tagged Alice User Image Bob aed4c Bob f93bae Alice aed4c Alice f93bae ByUser Global Secondary Index Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table Query FROM ImageTags-Index WHERE User=Alice Tagging
  • 62. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Recap: Image Tagging • Schema design on DynamoDB is all about the indexes • Local Secondary Indexes support flexible queries • Global Secondary Indexes unlock even more flexible queries • Many other use cases for indexing metadata on DynamoDB Tagging
  • 63. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Social Leaderboard materialized views Leaderboards
  • 64. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Friends Leaderboard • Find friends playing the same game • See where you stack up against your friends Leaderboards
  • 65. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Friends Leaderboard User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 Friends Table HighScores Table User Friend Bob Alice Bob Chuck Alice Bob Alice Chuck Alice Erin Leaderboards
  • 66. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Friends Leaderboard User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 Friends Table HighScores Table User Friend Bob Alice Bob Chuck Alice Bob Alice Chuck Alice Erin Alice Leaderboards
  • 67. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Friends Leaderboard User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 Friends Table HighScores Table User Friend Bob Alice Bob Chuck Alice Bob Alice Chuck Alice Erin 1. Query 2. BatchGetItemAlice Leaderboards
  • 68. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Inefficiencies in friends leaderboard • Always retrieve all friends – Some friends don’t play – Might care about top 10 scores only • Single-item operations for each friend – Retrieving 10 items individually is more expensive than 10 items in a row Leaderboards
  • 69. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Improved Friends Leaderboard: Reads User Friend H.S. Bob Alice 1000 Bob Chuck 20 Alice Bob 470 Alice Chuck 20 Alice Erin 580 User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 FriendScores Table HighScores Table Leaderboards
  • 70. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Improved Friends Leaderboard: Reads User Friend H.S. Bob Alice 1000 Bob Chuck 20 Alice Bob 470 Alice Chuck 20 Alice Erin 580 User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 User H.S. Friend Bob 20 Chuck Bob 1000 Alice Alice 20 Chuck Alice 470 Bob Alice 580 Erin Table Local Secondary Index FriendScores Table HighScores Table Leaderboards
  • 71. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Improved Friends Leaderboard: Reads User Friend H.S. Bob Alice 1000 Bob Chuck 20 Alice Bob 470 Alice Chuck 20 Alice Erin 580 User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 User H.S. Friend Bob 20 Chuck Bob 1000 Alice Alice 20 Chuck Alice 470 Bob Alice 580 Erin Table Local Secondary Index FriendScores Table HighScores Table Query FROM FriendScores-Index WHERE User=Alice, DESC, LIMIT 10Alice Leaderboards
  • 72. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture User HighScore Chuck 30 Alice 1000 Bob 470 Dave 850 Erin 580 Improved Friends Leaderboard: Writes User Friend H.S. Bob Alice 1000 Bob Chuck 30 Alice Bob 470 Alice Chuck 30 Alice Erin 580 FriendScores Table HighScores Table Chuck 1. UpdateItem 2. UpdateItem(s) Leaderboards
  • 73. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Recap: Friends Leaderboard • BatchGetItem helps retrieve many items • Query is cheaper than retrieving items individually • Trade off: reads can be made faster, cheaper, by doing more writes Leaderboards
  • 74. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture High-level use cases application architecture Architecture
  • 75. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture High-level use cases • Online game coordination • Real-time ad impression bids • Indexed event logging • Social messaging • Real-time voting • Real-time analytics Architecture
  • 76. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Online Game Coordination Availability Zone #1 DynamoDB region EC2 Availability Zone #1 EC2 EC2 CloudFron t Amazon S3 (content) (client app) (game servers) (lobby, stats service) Architecture
  • 77. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Online Game Coordination Availability Zone #1 DynamoDB region EC2 Availability Zone #1 EC2 EC2 1) Login 2) Request matchmaking 3) Get game server IP CloudFron t Amazon S3 Architecture
  • 78. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Online Game Coordination Availability Zone #1 DynamoDB region EC2 Availability Zone #1 EC2 EC2 1) Login 2) Request matchmaking 3) Get game server IP 4) Connect to server 5) Pull down assets 6) Other players join CloudFron t Amazon S3 Architecture
  • 79. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Online Game Coordination Availability Zone #1 DynamoDB region EC2 Availability Zone #1 EC2 EC2 1) Login 2) Request matchmaking 3) Get game server IP 4) Connect to server 5) Pull down assets 6) Other players join 7) Game ends 8) Update stats CloudFron t Amazon S3 Architecture
  • 80. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Indexed event logging User DateTime Product David 2014-06-22T21:01 Fire TV David 2014-06-22T21:02 Fire Phone David 2014-06-23T08:32 Fire Game Controller Alice 2014-06-23… … … … … Index webpage clicks by user Architecture
  • 81. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Real-time ad impression bids Architecture
  • 82. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Social networks, messaging Thread MessageId User Message Fri Night 21:01.1 David Hi! Fri Night 21:01.2 Steve What’s up? Fri Night 21:02.1 David Happy Hour? Participants Table, Secondary Index Messages Table Thread User Fri Night David Fri Night Steve DynamoDB? David User Thread David DynamoDB? David Fri Night Steve Fri Night Architecture
  • 83. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Real-time voting Voters AggregateVotes Table RawVotes Table Voting App Amazon Redshift Amazon EMR Architecture
  • 84. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Real-time analytics Kinesis-enabled app Amazon Kinesis DynamoDB Ingestion In-memory first-pass aggregation Durable aggregation Architecture
  • 85. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Real-time analytics Page Time Views index.html 2014-06-25T12:00 120 index.html 2014-06-25T12:01 122 index.html 2014-06-25T12:02 125 search.html 2014-06-25T12:00 200 PageViews Table Kinesis-enabled app UPDATE PageViews ADD 5 to Views WHERE Page=index.html AND Time=2014-06-25T12:00 Architecture
  • 86. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Real-time analytics Page Time Views index.html 2014-06-25T12:00 120 index.html 2014-06-25T12:01 122 index.html 2014-06-25T12:02 125 search.html 2014-06-25T12:00 200 PageViews Table Kinesis-enabled app Query PageViews WHERE Page=index.html AND Time >= 2014-06-25T12:00 LIMIT 10 Architecture
  • 87. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Recap • What is DynamoDB? • In-depth application modeling – Website Session Data (basic data model) – Tic Tac Toe (conditional writes) – Image Tagging (indexes and querying) – Gaming Leaderboard (materialized views) • High-level use cases – Application architecture, data models
  • 88. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Thanks! aws.amazon.com/dynamodb/developer-resources