Scale Your App for the Holidays with Amazon DynamoDB

Scale Your App for the Holidays with DynamoDB David Pearson, Business Development Manager build high scale applications in days fast | scalable | cost-effective© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

What Is NoSQL? CAP theorem BASE Not Only SQL schema-free horizontally scalable key-value eventually consistent unstructured open-source non-relational distributed© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

scalability big data challenge: RDBMS the cost of scalability infrastructure scaling scale-up = + bigger servers application scalingversatile scale-out = more servers change application to use…  read slaves  read caches  data shards© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

scalability NoSQL RDBMS infrastructure scaling = application scalingversatile specialized scale-out without application changes© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

General Characteristicsscalability RDBMS complexity © 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

NoSQL @ Amazona story in three parts…© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

key-value accessNoSQL @ Amazon complex queries transactionspart one - early days… analytics RDBMS used for “all kinds of access patterns” data (re-)partitioning bigger hardware is tempting versatile scalability easy to learn availability easy to query trade-off with consistency© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

specialized technologyNoSQL @ Amazon limited query capabilities simpler consistencypart two - dynamo… replicated DHT with consistency management • Consistent hashing • Optimistic replication • “Sloppy quorum” • Anti-entropy mechanisms • Object versioning© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

NoSQL @ Amazonpart two - dynamo… replicated DHT with consistency management eventual consistency higher availability no need to re-architect applications incremental scalability just add hardware! predictable performance simpler query model© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

NoSQL @ Amazonpart two - dynamo… replicated DHT with consistency management much better, but… higher availability no strong consistency still required developers to incremental scalability scaling effort required be administrators… predictable performance operational complexity install, patch, upgrade, balance clusters© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

NoSQL @ Amazon nosql database servicepart three - Amazon DynamoDB…  Fast & Predictable Performance  Seamless Scalability  Zero Administration “Even though we have years of experience with large, complex NoSQL architectures, we are happy to be finally out of the business of managing it ourselves.” – Don MacAskill, CEO© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

consistent, disk-onlyNoSQL @ Amazon writes (not memory) continuous replicationDesign Philosophy across multiple AZ’s • No compromise on durability or availability for performance • Express scale needs in simpler terms, not servers and disks • Scale will be our problem and not our customers • Extremely easy to use with no administration • Provide consistently low latencies provisioned throughput backed by SSDs© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

November Traffic at Amazon.com Capacity needed before DynamoDB 76% = wasted hardwareActualtraffic © 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

November Traffic at Amazon.com Capacity needed before DynamoDB Capacity we can provision with DynamoDBActualtraffic © 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Case Study small objects move faster! metadata in DynamoDBStores user objects in cloud objects in S3Queries and object searches are list all the objects in my driveserved by DynamoDB find all my music albums“We were excited by how fast we were able to put DynamoDB into production and how muchdeveloper time we saved. In addition, DynamoDB lets us scale up and down easily by simply reserving increased throughput capacity when we need it and dialing it back when we don’t” - Russel Dicker, Amazon CloudDrive© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Cloud Drive Photos #customers XGoal = enable customers to see Cloud Drive #photos photos on their KindleNeed = low latency access to the metadata, at any scaleResult = lower total cost of ownership (tco) lower admin effort required to scale hardware costs + operational costs + future tco opportunity cost of feature development© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

KINDLE DynamoDB access pattern list all images for photos uploaded by a customerCLOUD DRIVE Process Process Process Process Process Cloud Drive App Process Cloud Drive Servers Metadata Service image thumbnails album cover artAWS links to S3 S3 DynamoDB image Thumbnails ﬁles Dynamo S3 Images URLs DB © 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Getting started with DynamoDB… Two decisions + three clicks = ready for use© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Two decisions + three clicks = ready for use© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Provisioned Throughput effectively assigns each table its own set of servers • Reserve throughput for each table • Set at table creation, increased and decreased via an API call $0.01 per hour for every 10 units of Write Capacity $0.01 per hour for every 50 units of Read Capacity =10 writes per second… $1.00 per GB-month of Storage =50 strongly consistentfree tier… reads per second…100MB storage + 5 writes/sec ... for items up to+ 10 reads/sec each month 1KB in size © 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Items are indexed by primary key single hash keys and composite keys hash + range key-value access = extreme performance© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

table = collection of items attribute (key value pair)item (max size 64k) = string, number, binarycollection of attributes deviceid = 21EC2020-3AEA-1069-A2DD-08002B30309D total = 25.00 deviceid = 74ED9134-3FEC-9902-E8BA-19733F49779C total = 50.00 hash key © 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

mapping 1:M relations userid = 100 date = 2012-10-24-09-00-10 total = 25.00 userid = 100 date = 2012-10-24-09-00-11 total = 50.00 hash key range key© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

DynamoDBAPIs CreateTable PutItem UpdateTable GetItem read and manage tables DeleteTable write items UpdateItem DescribeTable DeleteItem ListTables query specific Query BatchGetItem bulk select or items OR scan Scan BatchWriteItem update the full table© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Query PatternsAvailable for hash-and-range primary key tablesRetrieve all items by hash keyRange key conditions: ==, <, >, >=, <=, begins with, betweenCounts. Top and bottom n values. Paged responses© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Getting started with DynamoDB… Designing a Photo Store© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Photo Store use case usage patterns data design enable users to upload and view photos from any device© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Photo Store use case usage patterns data design upload photos view photos by time© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Photo Store use case usage patterns data design table photos hash userid data model is optimized for retrieval performance range timestamp + photoid attributes location, resolution, comments, tags, s3link tags enable search integration s3link points to raw files© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Photo Store use case usage patterns data design table photos Potential Enhancements… hash userid • Albums range timestamp + photoid • Search integration (CloudSearch) attributes location, resolution, • Global caching and delivery of comments, tags, s3link media from S3 (CloudFront)© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Extreme scalability… Enhancing relational DB performance… and cost© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Scaling Relational DB’s Infrastructure Scaling bigger hardware (scaling up) Read Scaling #1 read replicas (slaves) Read Scaling #2 (hot keys) read caching (memcached) Write Scaling data sharding© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Scaling Relational DB’s Infrastructure Scaling bigger hardware (scaling up) Read Scaling #1 read replicas (slaves) Read Scaling #2 (hot keys) read caching (memcached) Write Scaling data sharding application changes required© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Scaling Relational DB’s with DynamoDB move simple, high-scale workloads Identify Design the new Implement code candidates DynamoDB table changes in app maximize access performance hopefully, for the last timeTables with high transaction volume (esp writes)Primary key-only is preferred (one non-PK index ok)No dependencies (FK’s, triggers, procedures) © 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

DynamoDB Scalability @ HalfBrick Studios• Fruit Ninja Frenzy (facebook) 8M• Moved game data into DynamoDB• Grew from 1 million to 8 million active monthly users in two weeks “it’s really tough to quickly scale a normal database system to handle that kind of rapid increase in load” 1M - Glen Arrowsmith, Systems Architect© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

DynamoDB Scalability “Creating a table that can serve 100,000 writes/second is no more work than creating a table that can serve 10 writes/second” Werner Vogels, Amazon CTO© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Super Bowl promotion Weatherbug app – lightning detection & alerting for 6M phonesMillions of interactions overa relatively short period Extending to 40M users/monthBuilt the app in 3 days – from Developed and tested in weeksdesign to production-ready “1/20th of the cost of the traditional DB approach” © 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

DynamoDB • minimal development time and effort • consistently low latency Fast • effortless scaling to meet workload demand Scalable • Reduces DB costs and increases reliability Cost-Effective • Free tier reduces initial development costs © 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Recommended Resources• Building Applications with DynamoDB (Matt Wood)  google search = youtube "building applications with dynamodb" http://www.youtube.com/watch?v=4jZthAFKAE4 http://www.slideshare.net/AmazonWebServices/building-applications-with-dynamodb• From the Super Bowl to WeatherBug (Werner Vogels) http://www.allthingsdistributed.com/2012/06/amazon-dynamodb-growth.html© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Questions? aws.amazon.com/dynamodb© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Scale Your App for the Holidays with Amazon DynamoDB

by Amazon Web Services on Jan 21, 2013

Accessibility

Upload Details

Usage Rights

Statistics

Scale Your App for the Holidays with Amazon DynamoDB Presentation Transcript