Downloaded 12 times
![© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data Modeling: SQL vs. NoSQL
Product Database
ID
Type
Price
Description
Products
ID
Author
Title
Category
Books
ID
Artist
Title
Genre
Albums
ID
Title
Category
Producer
Videos
ID
AlbumID
Title
Duration
Tracks
ID
Name
Bio
Actors
ActorID
VideoID
Actors
SQL
{
ProductID,
Type,
Price,
Description,
Author,
Title,
Fiction,
Category,
Date,
...
}
{
ProductID,
Type,
Price,
Description,
Artist,
Title,
Genre,
Tracks: [{
Title1,
Duration1
},
{
Title2,
Duration2
}]
...
}
{
ProductID,
Type,
Price,
Description,
Title,
Category,
Producer,
Director,
Actors: [{
ActorID,
Name,
Age,
ShortBio
}, ... ]
}
NoSQL
NoSQL design optimizes for
compute instead of storage.](https://image.slidesharecdn.com/2020-08-12awswebinar33-gettingstartedwithdatabasesonawsamazondynamodb-200811130630/85/AWS-SSA-Webinar-33-Getting-started-with-databases-on-AWS-Amazon-DynamoDB-10-320.jpg)
![© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Availability Zone A
Partition A
Host 4 Host 6
Availability Zone B Availability Zone C
Partition APartition A Partition CPartition C Partition C
Host 5
Partition B
Host 1 Host 3Host 2
Partition B
Host 7 Host 9Host 8
Partition B
CustomerOrdersTable
Data is always
replicated to three
Availability Zones
3-way replication
OrderId: 1
CustomerId: 1
ASIN: [B00X4WHP5E]
Hash(1) = 7B
Highly Available And Durable
Partition A](https://image.slidesharecdn.com/2020-08-12awswebinar33-gettingstartedwithdatabasesonawsamazondynamodb-200811130630/85/AWS-SSA-Webinar-33-Getting-started-with-databases-on-AWS-Amazon-DynamoDB-23-320.jpg)
Uploaded byCobus Bernard
AWS SSA Webinar 33 - Getting started with databases on AWS Amazon DynamoDB
The document is a comprehensive guide on Amazon DynamoDB, emphasizing its features, use cases, and advantages over traditional relational databases. It highlights aspects like scalability, cost-effectiveness, durability, and security, while also providing technical insights into data modeling and partitioning. Additionally, it covers the integration with the AWS ecosystem and the operational benefits of using DynamoDB for large-scale applications.
More Related Content
Similar to AWS SSA Webinar 33 - Getting started with databases on AWS Amazon DynamoDB
More from Cobus Bernard
AWS SSA Webinar 33 - Getting started with databases on AWS Amazon DynamoDB
- 1. Cobus Bernard Sr DeveloperAdvocate Amazon Web Services GettingStartedwith DatabasesonAWS: Amazon DynamoDB @cobusbernard cobusbernard cobusbernard CobusCloud
- 2. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Agenda NoSQL Basics Amazon DynamoDB Tables & Indices Customer Use Cases Additional Concepts Q & A
- 3. © 2020, AmazonWeb Services, Inc. or its Affiliates. NoSQL Basics
- 4. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Data volume since 2010 • 90% of stored data generated in last few years • 1 terabyte of data in 2010 equals 40 petabytes in 2019 • Linear correlation between data pressure and technical innovation • No reason these trends will not continue over time
- 5. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Not Only SQL (NoSQL) 0000 {“Texas”} 0001 {“Illinois”} 0002 {“Oregon”} TXW A I L Key Column 0000-0000-0000-0001 Game Heroes Version 3.4 CRC ADE4 Key Value GraphDocument Column-family Amazon’s Highly Available Key-value Store Jan 2012Fall 2007 Late 2007 Amazon SimpleDB Amazon DynamoDBDynamo Nov 2012 Amazon RedShift Amazon Neptune May 2019 Amazon DocumentDB Jan 2019
- 6. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Scaling relational vs. non-relational databases Traditional SQL NoSQL DB DB Scale up by moving to a larger instance size Scale out easily by adding more partitions
- 7. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Technical Challenges in Scaling Traditional Relational Database • Moving to a larger instance is not trivial • A larger instance must exist • Sharding is required after a certain point • Amazon RDS minimizes these pains, but not entirely
- 8. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Scaling NoSQL • Good partitioning scheme affords even distribution of both data and workload, as they grow • Key concept: partition key • Ideal scaling conditions: 1. Partition key is from a high cardinality set (that grows) 2. Requests are evenly spread over the key space 3. Requests are evenly spread over time
- 9. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Good Partitioning Partition K 2000 RCUs Partition M 2000 RCUs Partition A Shoppers A B C D Partition B Partition C Partition D Try to avoid hot partitions!! Ideally traffic should be distributed evenly across partitions
- 10. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Data Modeling: SQL vs. NoSQL Product Database ID Type Price Description Products ID Author Title Category Books ID Artist Title Genre Albums ID Title Category Producer Videos ID AlbumID Title Duration Tracks ID Name Bio Actors ActorID VideoID Actors SQL { ProductID, Type, Price, Description, Author, Title, Fiction, Category, Date, ... } { ProductID, Type, Price, Description, Artist, Title, Genre, Tracks: [{ Title1, Duration1 }, { Title2, Duration2 }] ... } { ProductID, Type, Price, Description, Title, Category, Producer, Director, Actors: [{ ActorID, Name, Age, ShortBio }, ... ] } NoSQL NoSQL design optimizes for compute instead of storage.
- 11. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Why NoSQL? • Massive scale – At an affordable price • Predictable, low latency – Regardless of the scale or load • Flexible schema – e.g. DynamoDB: Key-value pairs and JSON documents stored in the same table do not need to be identical in form Why not NoSQL? • If you need object relations or joins • If you need flexible, ad hoc queries – Use SQL databases
- 12. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. SQL NoSQL Optimized for storage Optimized for compute Normalized/relational Denormalized/hierarchical Ad-hoc queries Instantiated views Scale vertically Scale horizontally Good for traditional OLTP Handle unstructured/semi-structured data at scale SQL vs NoSQL?
- 13. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Use Cases Market orders Tokenization (PHI, credit cards) Chat messages User profiles IoT sensor data & device status File metadataSocial media feeds Shopping cart Sessions
- 14. © 2020, AmazonWeb Services, Inc. or its Affiliates. Amazon DynamoDB
- 15. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Highly available and durable Consistently fast at any scaleFully managed Secure Integrates with AWS Lambda, Amazon Redshift, and more Amazon DynamoDB Cost-effective
- 16. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Single tables serving… Millions of requests per second Trillions of items Hundreds of terabytes of storage
- 17. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Fully Managed Service = Automated Operations Scaling High Availability Database backups DB software patches DB software installs OS patches Server maintenance Rack and stack Power, HVAC, net OS Installation App Optimization DB hosted on premises you
- 18. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Fully Managed Service = Automated Operations you DB hosted on premises DB hosted Amazon EC2 you Scaling High Availability Database backups DB software patches DB software installs OS patches Server maintenance Rack and stack Power, HVAC, net OS Installation App Optimization Server maintenance Rack and stack Power, HVAC, net OS Installation Scaling High Availability Database backups DB software patches DB software installs OS patches App Optimization
- 19. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Fully Managed Service = Automated Operations you DB hosted on premises DynamoDB you Scaling High Availability Database backups DB software patches DB software installs OS patches Server maintenance Rack and stack Power, HVAC, net OS Installation App Optimization Scaling High Availability Database backups DB software patches DB software installs OS patches Server maintenance Rack and stack Power, HVAC, net OS Installation App Optimization
- 20. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Consistently low latency at scale PREDICTABLE PERFORMANCE! ConsistentSingle-Digit Millisecond Latency Requests(millions) Latency(milliseconds)
- 21. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Durability
- 22. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Highly Available And Durable Designed to support 99.99% of availability Built for high durability WRITES 3-way replication Persisted to disk (Custom SSD) READS Strongly or eventually consistent No latency trade-off Data is always replicated to three Availability Zones
- 23. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Availability Zone A Partition A Host 4 Host 6 Availability Zone B Availability Zone C Partition APartition A Partition CPartition C Partition C Host 5 Partition B Host 1 Host 3Host 2 Partition B Host 7 Host 9Host 8 Partition B CustomerOrdersTable Data is always replicated to three Availability Zones 3-way replication OrderId: 1 CustomerId: 1 ASIN: [B00X4WHP5E] Hash(1) = 7B Highly Available And Durable Partition A
- 24. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon DynamoDB – Backup and Restore Only cloud database to provide on demand and continuous backups Point in time restore for short term retention and data corruption protection (35 days) Point in time recovery with restore times in a few hours depending on table size On-demand backups for long-term data archival and compliance
- 25. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Security
- 26. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Secure Fully integrated with AWS Identity and Access Management (IAM) for authentication and access control. Provides fine-grained access control at a table, item or attribute level. Integrated with AWS CloudTrail to capture changes to DynamoDB configuration and table setup. Integrated with AWS CloudWatch to measure metrics around DynamoDB performance and set alarms to track specific events.
- 27. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Amazon Virtual Private Cloud (VPC) Endpoints • Access DynamoDB via secure Amazon VPC endpoint • Control access to tables via each VPC endpoint with unique IAM roles and permissions Features Key Benefits • Turn off access from public internet gateways enhancing privacy and security • Fast, secure data transfer between Amazon VPC and DynamoDB
- 28. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Encryption at Rest Server- side encryptio n Support compliance certification s No application code rewrites
- 29. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Performance
- 30. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Consistent, High Performance Request Volume Latency
- 31. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. • Dynamically adjusts partition capacity based on real-time traffic • To better handle imbalanced workloads • Best practices for even load distribution still matter • Reduces throttling • As long as there is enough capacity provisioned for the table • Up to the hard limit of partition write capacity • Triggered by throttling Now instantaneous!!! Adaptive Capacity
- 32. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Adaptive Capacity
- 33. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Real world impact Background operations don’t limit burst anymore
- 34. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. DynamoDB Accelerator (DAX) • Fully managed, highly available: handles all software management, fault tolerant, replication across multi-AZs within a region • DynamoDB API compatible: seamlessly caches DynamoDB API calls, no application re-writes required • Write-through: DAX handles caching for writes • Flexible: Configure DAX for one table or many • Scalable: scales-out to any workload with up to 9 read replicas • Manageability: fully integrated AWS service: Amazon CloudWatch, Tagging for DynamoDB, AWS Console • Security: Amazon VPC, AWS IAM, AWS CloudTrail, AWS Organizations Features DynamoDB Your Applications DynamoDB Accelerator Table #1 Table #2
- 35. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. DynamoDB Accelerator (DAX) Milliseconds to microseconds In-Memory performance and throughput More than an order of magnitude faster!
- 36. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Scalability
- 37. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Scaling NoSQL Throughput - Provision capacity on demand, as needed - WCU: Write capacity per second (up to 1KB) - RCU: Read capacity per second (up to 4KB) - Partitions are scaled out horizontally to automatically cover the throughput requirements of the workload Size - Add any number of items to a table (Max item size is 400 KB)
- 38. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. ScalingThroughput: Auto Scaling $$$ Savings
- 39. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. ScalingThroughput: Auto Scaling • Fully-managed, automatic • Scale up when you need it • Scale down when you don’t • On by default • Scheduled Auto Scaling
- 40. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Auto Scaling
- 41. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. With Auto Scaling Without Auto Scaling • Remove the guesswork out of provisioning adequate capacity • Increases capacity as application requests increase, ensuring performance • Decreases capacity as application requests reduce, reducing costs • Full visibility into scaling activities from console Key Benefits Scaling Throughput: Auto Scaling
- 42. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. ScalingThroughput: On-demand Start at Zero No Limit Features • No capacity planning, provisioning, or reservations– simply make API calls • Pay only for the reads and writes you perform • Instantly accommodates your workload as traffic ramps up or down
- 43. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Govern Max Consumption Auto Scaling Provisioned Set a Minimum Start at Zero No Limit On-demand ScalingThroughput: On-demand
- 44. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Availability
- 45. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. DynamoDB GlobalTables F u l l y m a n a g e d , m u l t i - p r i m a r y, m u l t i - r e g i o n d a t a b a s e Build high performance, globally distributed applications Low latency reads & writes to locally available tables Disaster proof with multi-region redundancy Easy to setup and no application re-writes required
- 46. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Cost-effective - Perpetual free tier: - 25GB of storage - 25 WCUs and 25 RCUs of provisioned capacity - 2.5 million read requests from DynamoDB Streams - 25 rWCUs for global tables deployed in two AWS Regions - Pay-as-you-grow for capacity and storage independently - Auto scaling (Target utilization at 75%) - Time-to-live (TTL) - Automatically purges data at no extra charge - (Deleting tables doesn’t incur charges either) - Cost Allocation Tagging
- 47. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Integrated: DynamoDB + AWS Ecosystem Amazon DynamoDBAmazon S3 Amazon Kinesis Amazon EMR Amazon Redshift AWS Data Pipeline AWS Mobile Hub AWS Lambda Amazon ES Amazon SNS Amazon CloudWatch Amazon IOT
- 48. © 2020, AmazonWeb Services, Inc. or its Affiliates. Tables & Indices
- 49. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Partition Key Mandatory Key-value access pattern Determines data distribution Optional Model 1:N relationships Enables rich query capabilities DynamoDBTable A1 (partition key) A2 (sort key) A3 A4 A7 A1 (partition key) A2 (sort key) A6 A4 A5 A1 (partition key) A2 (sort key) A1 (partition key) A2 (sort key) A3 A4 A5 Sort Key Table Items All items for key ==, <, >, >=, <= “begins with” “between” “contains” “in” sorted results counts top/bottom N values
- 50. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Local secondary index (LSI) Alternate sort key attribute Index is local to a partition key A1 (partition) A3 (sort) A2 (item key) A1 (partition) A2 (sort) A3 A4 A5 LSIs A1 (partition) A4 (sort) A2 (item key) A3 (projected) Table KEYS_ONLY INCLUDE A3 A1 (partition) A5 (sort) A2 (item key) A3 (projected) A4 (projected) ALL 10 GB maximum per partition key; LSIs limit the number of range keys!
- 51. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Global Secondary Index (GSI) Alternate partition and/or sort key Index is across all partition keys A1 (partition) A2 A3 A4 A5 GSIs A5 (partition) A4 (sort) A1 (item key) A3 (projected) Table INCLUDE A3 A4 (partition) A5 (sort) A1 (item key) A2 (projected) A3 (projected) ALL A2 (partition) A1 (itemkey) KEYS_ONLY Online indexing Read capacity units (RCUs) and write capacity units (WCUs) are provisioned separately for GSIs
- 52. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. How do GSI updates work? Table Primary table Primary table Primary table Primary table Global secondary index Client 2. Asynchronous update (in progress) If GSIs don’t have enough write capacity, table writes are throttled!
- 53. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. LSI or GSI? LSI GSI Create at table creation Create any time Shares WCU/RCU with table WCU/RCU independent of table Size <= 10GB* No size limits Limit = 5 Limit = 20 Strong Consistency Eventual Consistency *10GB size limit is for a item collection size with an LSI
- 54. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Data types Type DynamoDB Type String String Integer, Float Number Timestamp Number or String Blob Binary Boolean Bool Null Null List List Set Set of String, Number, or Binary Map Map
- 55. © 2020, AmazonWeb Services, Inc. or its Affiliates. Customer Use Cases
- 56. © 2019, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Scaling high-velocity use cases with DynamoDB
- 57. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. RDBMS DynamoDB Amazon’s Path to DynamoDB
- 58. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Prime Day 2019: 45.4M requests / second
- 59. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Migration from Cassandra: Backup & restore on mobile app for 300M users Almost 1 PB in DynamoDB, 130M daily API requests Migrated from Cassandra to DynamoDB Consistent performance and 70% cost savings (TCO) DynamoDB provided consistent high performance at a drastically lower cost than Cassandra.” Seongkyu Kim Samsung “
- 60. © 2020, AmazonWeb Services, Inc. or its Affiliates. Additional Concepts
- 61. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Time-to-live (TTL) ID Name Size Expiry 1234 A 100 1456702305 2222 B 240 1456702400 3423 C 150 1459207905 • Automatically delete items from a table based on expiration timestamp • User defined TTL attribute in epoch time format • TTL activity recorded in DynamoDB Streams TTL Attribute Features Key Benefits • Reduce costs by deleting items no longer needed • Optimize application performance by controlling table size growth • Trigger custom workflows with Streams and Lambda
- 62. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Cost allocation tagging
- 63. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Cost allocation tagging • Track costs: AWS bills broken down by tags in detailed monthly bills and Cost Explorer • Flexible: Add customizable tags to tables, indexes and DAX clusters Features Key Benefits • Transparency: know exactly how much your DynamoDB resources cost • Consistent: report of spend across AWS services
- 64. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. DynamoDB Transactions Single API Call Simplify your code by executing multiple, all-or-nothing actions within and across tables with a single API call.
- 65. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Key Features • Provides atomicity, consistency, isolation, and durability (ACID) in DynamoDB. • You can perform transactions both within and across multiple DynamoDB tables. • Native, server-side solution that provides better performance and lower costs than client-side libraries.
- 66. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. NoSQL Workbench for DynamoDB • A graphical user interface to visualize data and perform DynamoDB operations. • Easily build new data models from scratch, or import and modify existing data models. • Visualize data models based on their applications' access patterns. • Perform data-plane operations and generate sample code in multiple languages.
- 67. © 2020, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Getting Started DynamoDB Local Document SDKs DynamoDB Developer Resources https://aws.amazon.com/dynamodb/developer-resources/ https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.SDKOverview.html https://amazon-dynamodb-labs.com/
- 68. © 2020, AmazonWeb Services, Inc. or its Affiliates. twitch.tv/aws – Mo/Fr @ 11am SAST Bean Streaming twitch.tv/aws – Thu @ 12:00 SAST AWS Africa Office Hours youtube.com/c/CobusCloud bit.ly/notC_notD (Watch the recorded weekly sessions)
- 69. Thank you! © 2020,Amazon Web Services, Inc. or its affiliates. All rights reserved. Cobus Bernard Sr Developer Advocate Amazon Web Services @cobusbernard cobusbernard cobusbernard CobusCloud
Editor's Notes
- #5 To fully appreciate the need for NoSQL… Let’s start by looking into how much data volume has grown in the last few years. 90% of data was generated in the last few years. 1 TB vs 40 PB .. To put that into perspective… We are starting to see Businesses with multi-TB have exploded to multiPB databases. As data volume increased, we started innovating data processing systems that would scale to process the large volume of data https://techjury.net/stats-about/big-data-statistics/
- #6 Note to presenters : Intent of this slide idea is to convey two different ideas : Variety of databases with need for different data types 2. AWS playing active role in this rapidly changing industry. Use the animation in the order included along with the notes below. =============================================================== Different forms of data. Beyond relational data that can be handled by the traditional relational databases. We have data that can be represented as Key-Value pairs, connected data that is best represented as a Graph, Documents, columnar data that is best suited for aggregations, so on and so forth. If you observe AWS, our database services have evolved according to the different types of data needs in the industry. Dynamo was proposed as a key-value store back in Fall 2007. Later that year, we released SimpleDB, which is a managed service for NoSQL databases. A few years down the line, by Jan 2012, AWS released DynamoDB, which is again a NoSQL database service offering, that offers seamless scaling and single digit milli second latency. We recently at the end of 2017 released in preview mode Neptune, which is managed service for graph databases. As you can see we are constantly improving our database managed services offerings to meet industry demand and lead data type trends. Today we will focus on DynamoDB, which is our enterprise level managed NoSQL offering.
- #7 Let’s talk scaling database. Relational - Data is normalized. To enable joins, You are tied to a single partition and a single system. performance on the hardware specs of the primary server. To improve performance, Optimize -- Move to a bigger box. You may still run out of headroom. Create Read Replicas. You will still run out. Scale UP. NoSQL -- NoSQL databases were designed specifically to overcome scalability issues. Scale “out” data using distributed clusters, low-cost hardware, throughput + low latency Therefore, Using NoSQL, businesses can scale virtually without limit. Explain partitions here
- #9 Use high-cardinality attributes. These are attributes that have distinct values for each item, like email, employeeid, customerid, sessionid, orderid, and so on. Use composite attributes. Try to combine more than one attribute to form a unique key, if that meets your access pattern. For example, consider an orders table with customerid+productid+countrycode as the partition key and order_date as the sort key.
- #10 Items are stored based on partition key. Avoid hot partitions keys so your requests are distributed across partitions/shards. While adaptive capacity will enable you to continue writing to a partition without being throttled (as long as you are under the table limit), throttling can occur if a single partition receives more than 3,000 RCUs or 1,000 WCUs.
- #11 Generic product catalog. Table relationships in normalized. A product could be a book – say the Harry Potter Series. There’s a 1:1 relationship. Or it could be a movie.. You can imagine the types of queries that you’d have to execute. 1. Show me all the movies starring. 2. the entire product catalog. This is Resource intensive – perform complex join ** NoSQL you have to ask – how will the application access the data? optimize for the costlier asset. No joins. Just a select. Hierarchical structures. Designed by keeping in mind Access patterns. Via duplication of data (storage), optimized for compute, it is fast.
- #12 Businesses are starting to see scalability problems with relational databases. I once had a customer say they top out with relational at around 3,000 requests per second and had to scale up to move to bigger hardware. With NoSQL, we have a technology that can easily sale to 100s of nodes, or even 1000s, and the scalability bottleneck goes away. Excellent for OLTP applications that scale, real time data access, fast, low latency, user cannot wait. == They store data in a de-normalized hierarchical view, that makes it faster and easier to access the data.
- #13 Businesses are starting to see scalability problems with relational databases. I once had a customer say they top out with relational at around 3,000 requests per second and had to scale up to move to bigger hardware. With NoSQL, we have a technology that can easily sale to 100s of nodes, or even 1000s, and the scalability bottleneck goes away. Excellent for applications that scale, real time data access, fast, low latency, user cannot wait. == They store data in a denormalized hierarchical view, that makes it faster and easier to access the data. SQL is good for OLAP (and maybe HTAP), and NoSQL (at least for DynamoDB) is best for OLTP at any scale. DynamoDB's completely serverless natures and decoupling of compute makes it a great choice for tiny workloads and massive workloads, with incredibly elastic scaling between the extremes.
- #14 We are seeing DynamoDB being used in several types of applications and workloads. Here are some possible sample usecases. And we will look at a few of our customer workloads in the next few slides. If you look at these usecases, you will see that they have two things in common : they all need low latency and they need to be able to scale seamlessly. Take for example : Chat messages – A messenger application needs to provide real time experience to the end users and need to be able to accommodate huge volumes. Very similar to IoT Sensor data : accommodate huge volumes of data from the sensor and make the data available for real-time needs. Same with social media feeds. Let’s look into basics of DynamoDB next and then we will see how some of our customers are taking advantage of DynamoDB capabilities to build some of these workloads.
- #16 With DynamoDB, what are you getting is : -- Fully managed service that you can start using with just a few clicks on AWS console -- Ability to create a table that is highly scalable and gives you consistent performance at any scale. -- Highly available and durable -- Secure : Access control using fine grained policies -- Integrates with other AWS services like lambda, redshift enabling you to architect applications that automatically react to data changes. -- As with other AWS services, only pay for what you use. Key take away: Using DynamoDB customers get consistent, single-digit millisecond latency at any scale. Now lets take a look at these capabilities in detail. ============== Only notes for speaker, don’t say the below at this time ================ DynamoDB supports both document and key-value store models, and offers a range of features including global secondary indexes, fine-grained access control via AWS Identity and Access Management, support for event-driven programming, and more. == Fast, Consistent Performance Amazon DynamoDB is designed to deliver consistent, fast performance at any scale for all applications. Average service-side latencies are typically single-digit milliseconds. As your data volumes grow and application performance demands increase, Amazon DynamoDB uses automatic partitioning and SSD technologies to meet your throughput requirements and deliver low latencies at any scale. Fully Managed Amazon DynamoDB is a fully managed cloud NoSQL database service – you simply create a database table, set your throughput, and let the service handle the rest. You no longer need to worry about database management tasks such as hardware or software provisioning, setup and configuration, software patching, operating a reliable, distributed database cluster, or partitioning data over multiple instances as you scale. Flexible Amazon DynamoDB supports both document and key-value data structures, giving you the flexibility to design the best architecture that is optimal for your application. Highly Scalable When creating a table, simply specify how much request capacity you require. If your throughput requirements change, simply update your table's request capacity using the AWS Management Console or the Amazon DynamoDB APIs. Amazon DynamoDB manages all the scaling behind the scenes, and you are still able to achieve your prior throughput levels while scaling is underway. Event Driven Programming Amazon DynamoDB integrates with AWS Lambda to provide Triggers which enables you to architect applications that automatically react to data changes. Fine-grained Access Control Amazon DynamoDB integrates with AWS Identity and Access Management (IAM) for fine-grained access control for users within your organization. You can assign unique security credentials to each user and control each user's access to services and resources.
- #17 Those if you who are involved in spinning up and managing your own servers surely realize how resource intensive it is to manage your own infrastructure. It can be possible to underestimate the cost and complexity of maintaining…. You have to think about power, cooling, OS maintenance and patching. Now imagine managing a 1000 node cluster, this can become very resource intensive Amazon EC2 is an AWS service for is the compute capacity in cloud, it is resizable. Database instance hosted in an EC2 instance takes away some of the overhead. But, you still need to think about scalability and availability.
- #18 Those if you who are involved in spinning up and managing your own servers surely realize how resource intensive it is to manage your own infrastructure. It can be possible to underestimate the cost and complexity of maintaining…. You have to think about power, cooling, OS maintenance and patching. Now imagine managing a 1000 node cluster, this can become very resource intensive Amazon EC2 is an AWS service for is the compute capacity in cloud, it is resizable. Database instance hosted in an EC2 instance takes away some of the overhead. But, you still need to think about scalability and availability.
- #19 Those if you who are involved in spinning up and managing your own servers surely realize how resource intensive it is to manage your own infrastructure. It can be possible to underestimate the cost and complexity of maintaining…. You have to think about power, cooling, OS maintenance and patching. Now imagine managing a 1000 node cluster, this can become very resource intensive Amazon EC2 is an AWS service for is the compute capacity in cloud, it is resizable. Database instance hosted in an EC2 instance takes away some of the overhead. But, you still need to think about scalability and availability.
- #20 This is the value that is built into DynamoDB. With DynamoDB, you get an easy-to-use database. You don’t have to spin up any servers. You can easily design serverless scalable applications with DynamoDB. You get scalability and multi-AZ replication without designing a distributed system. You get ongoing security upgrades, software improvements, cost reduction efforts, monitoring…without any effort at all. DDB is fully managed service, you have all of that benefit built into it. We built Dynamo to just work so you can focus on your app.
- #21 In any business, as your business scales up, you need a way to easy scale to meet the traffic, and be able to get consistent predictable latency at any scale. You need a way to scale down as your business needs changes. DynamoDB was designed to offer consistent and predictable single-digit millisecond latency, at any scale. And you only pay for what you use. NO limit on throughput. No limit on Size – PB of data any number of items. The latency characteristics of DynamoDB are under 10 milliseconds and highly consistent. Most importantly, the data is durable in DynamoDB, constantly replicated across multiple data centers and persisted to SSD storage. Predictable Performance This is obviously something that’s important and valuable in any industry, whether it’s powering the New York Times recommendation engine, storing and retrieving game data for the game Fruit Ninja, or powering queries and fast data retrieval for Major League Baseball Advanced Media. Predictable performance at scale is a must-have for many web apps, and DynamoDB was designed specifically to deliver on this.
- #23 DynamoDB is built for high availability and 99.99% durability. All ”Writes” are persisted on to a SSD disk and replicated to 3 availability zones. Reads can be configured to be “strong” or ”eventual” consistent. There is no latency tradeoff with either configuration, however the read capacity is used is different. We will talk about read and write capacity units in a few slides. Now Let’s see how the 3 – way replication looks.
- #24 Here we are inserting an item into”CustomerOrders” table. CustomerId value, actually the hash of the customerId value is sent to three different availability zones.
- #25 DynamoDB can back up your data with per-second granularity and restore to any single second from the time PITR was enabled up to the prior 35 days
- #32 EMPHASIZE: COMPLETELY AUTOMATED
- #33 The following diagram illustrates how adaptive capacity works. The example table is provisioned with 400 write-capacity units (WCUs) evenly shared across four partitions, allowing each partition to sustain up to 100 WCUs per second. Partitions 1, 2, and 3 each receive write traffic of 50 WCU/sec. Partition 4 receives 150 WCU/sec. This hot partition can accept write traffic while it still has unused burst capacity, but eventually it will throttle traffic that exceeds 100 WCU/sec. DynamoDB adaptive capacity responds by increasing partition 4's capacity so that it can sustain the higher workload of 150 WCU/sec without being throttled. This feature used to have a delay before kicking in, but it is now also instantaneous since the May 23rd, 2019 update. https://aws.amazon.com/about-aws/whats-new/2019/05/amazon-dynamodb-adaptive-capacity-is-now-instant/
- #34 99.9% availability Talk about how there are no background ops included in burst cap calculation In almost all cases, only exceeding partition MAX IOPS will lead to throttle (or extended overuse of burst capacity)
- #35 Node, .NET, Python, Java SDKs with Go in the works
- #38 At table creation time you can specifiy : Keys : primary/sort WCU : write capacity unit (1 KB/sec) RCU : read capacity unit (4 KB/sec) Size of the table automatically increases as you add more items. There is a limit of item size at 400KB. This is a hard limit As more items are added as size increases, the table is partitioned automatically for you. Size and provisioning capacity of the table are equally distributed for all partitions. And new partitions are added when either the capacity or size exceeds the formula above.
- #41 Here’s a screen shot that shows how you can configure Dynamo DB auto scaling from the console. The same functionality is available through cli and SDK as well. You can choose to auto-scale either one or both of them. Similar to min/max instances in an auto scaling group, you can specify minimum and maximum read/write capacity. Additionally, you can chose to apply the same settings to global secondary indexes. And of course, you have complete control to grant DynamoDB permission to scale the provisioned capacity on your behalf. You do this through IAM role. You can chose an existing role or create a new role that allows Dynamo scaling operations.
- #42 If you are familiar with auto scaling of EC2s, this is very similar to that. You take advantage of auto scaling to provision capacity when your application needs demand it while avoiding unnecessary over provisioning. While you control the number of instances in a fleet of EC2 instances (an autoscaling group) in the first case, here for DynamoDB, the two knobs you are controlling are “Read capacity” and “Write capacity”. Completely independent of each other and based on target utilizations set. As a managed and automatic feature, this takes out the guesswork out of provisioning capacity. Lets see how it looks when done from the console.
- #44 On-demand instances can scale much faster than autoscaling DynamoDB. Autoscaling rules monitor utilization of the table and scales similar to EC2 instances. This can take minutes. For spikey and unpredictable workloads, we suggest using on-demand scaling for DynamoDB.
- #47 Dynamo is very cost effective. Every month you get 25GB, 200 million transactions (i.e., 25 writes and 25 reads per second), 2.5 million read requests from DynamoDB Streams and Global tables upto 2 regions for free. EVERY MONTH, not just the first 12 months of signing up. You get on demand capacity and storage independently. Additionally, you can get benefit of auto scaling for your provisioned throughput, so you are not paying for unused provisioning capacity. TTL is a concept that will effectively help you manage your table size without having to pay item deletion charges. And lastly, we have cost allocation tagging, that allows you to keep track of dynamo related expenses including costs for tables, indexes, global tabels etc. So you can keep tabs on your expenses for a given project or for a given team etc. I have a few additional slides about auto scaling, TTL and cost tagging that we will cover in a little bit.
- #50 Data is stored in Tables. Think of Table as the “Database” Within table we have “Items” We have our first item. Item has attributes : 5 in this case. Now lets see what is the next item is. Here you can see there are only two attributes. As we add more items to the table, we can see that Attributes can vary between the items, Each item can have a different set of attributes than the other items. (as with any NoSQL database). You can also see the primary key or the partition key. It uniquely identifies each item. Also determines HOW DATA IS Partitioned STORED. Partition key is mandatory. Optional Sort key – you have a composite key; Sort keys help to create 1:many relationships, and useful in range queries. Lets consider an orders table. Partition key could be order_id, and sort_key could be ‘customer_id’
- #51 Some times you need to query data using the primary key; But sometimes you might have a need to query by an attribute that is not your primary/seconday key. Lets say we want to find out all fullfiled orders. We would have to query for all orders and then look for fullfilled ones in the results. Not very efficient with large tables. But we have “LSIs” to help us out. We can create an “LSI” with the same primary key (order_id) and a different secondary key (fullfilled). Now you query can be based on the key of the LSI. Fast and efficient. LSI is collocated on the same partition as the item in the table, so this gives us consistency. When an item is updated, LSI is updated, and then ack’d. LSI is partitioned by the same primary key as the parent table. Different Sort key. In the Index, you can choose to have just the keys, or other attributes projected or included all attributes – Depending on what attributes you want returned with the query. There is a limit of 10GB on LSI storage. Note that LSIs are local to partition key. And LSIs are using the RCU/WCU of the original table.
- #52 Taking the concept of indexes a little further. Some applications might need to perform many kinds of queries, using a variety of different attributes as query criteria. Doesn’t fit into either using existing primary key/sort key. In this case you define a GSI. Global Secondary Indexes – Parallel tables or secondary tables. GSI can have a partition key that is different from the Table. They can also have an alternate sort key. Customers, Orders, Date Range. Partition by Order Id and query for a date range. Note: When you create a GSI, you must specify read and write capacity units for the expected workload on that index. Similar to an LSI, you can choose to have just the keys, or other attributes projected or included all attributes – Depending on what attributes you want returned with the query.
- #53 Think of this as a parallel table asynchronously populated by DynamoDB. Eventually consistent. GSI updates typically happen within a second. Throughput for GSI is important.. That is important on how soon the GSI will be updated. Note: When you create a GSI, you must specify read and write capacity units for the expected workload on that index. 1 Table update = 0, 1 or 2 GSI updates
- #54 Customers often ask if LSI should be used or GSI. When should you use GSI vs LSI?? More flexibility with GSI. With a local secondary index, there is a limit on item collection sizes: For every distinct partition key value, the total sizes of all table and index items cannot exceed 10 GB You can have only 5 LSI and 20 GSI, however, with GSI, you have the flexibility to create them after the table is created. LSI must be created when the table is defined. LSI can be modeled as a GSI If data size in an item collection > 10 GB (Example, many orders for a customerID) use GSI that’s the only choice. Because LSI limit the data size in a particular partition. If eventual consistency is okay for your scenario, use GSI – it works for 99% of the scenarios out there.
- #57 Ad Tech AdRoll http://aws.amazon.com/solutions/case-studies/adroll/ DataXu http://info.qubole.com/how-dataxu-manages-big-data AdBrain http://www.adbrain.com/careers-generalapp/ DoApp https://aws.amazon.com/solutions/case-studies/doapp/ VidRoll https://aws.amazon.com/solutions/case-studies/vidroll/ Fiksu https://aws.amazon.com/solutions/case-studies/fiksu/ TubeMogul https://www.tubemogul.com/engineering/using-contextual-information-in-programmatic-advertising/ TCC https://github.com/TheClimateCorporation/mandolin Gaming Supercell http://aws.amazon.com/solutions/case-studies/supercell/ Zynga https://aws.amazon.com/solutions/case-studies/zynga/ Nexon http://aws.amazon.com/solutions/case-studies/nexon PennyPop http://aws.amazon.com/solutions/case-studies/battle-camp/ Frontier http://aws.amazon.com/solutions/case-studies/frontier-games/ scopely https://aws.amazon.com/solutions/case-studies/scopely/ Unalis https://aws.amazon.com/solutions/case-studies/unalis/ IoT MLBAM http://aws.amazon.com/solutions/case-studies/major-league-baseball-mlbam/ ACTi https://aws.amazon.com/solutions/case-studies/acti-case-study/ Canary https://aws.amazon.com/solutions/case-studies/canary/ Dropcam https://aws.amazon.com/solutions/case-studies/dropcam/ MediaTek https://aws.amazon.com/solutions/case-studies/mediatek/ Devicescape https://aws.amazon.com/solutions/case-studies/devicescape/ Mobile Duolingo http://aws.amazon.com/solutions/case-studies/duolingo-case-study-dynamodb/ Mapbox https://www.mapbox.com/blog/scaling-the-mapbox-infrastructure-with-dynamodb-streams/ Redfin http://aws.amazon.com/solutions/case-studies/redfin/ and https://www.youtube.com/watch?v=YiaPjILR9zw Remind https://aws.amazon.com/solutions/case-studies/remind/ Infraware http://aws.amazon.com/solutions/case-studies/infraware/ Myriad http://aws.amazon.com/solutions/case-studies/myriad-group/ Peak http://aws.amazon.com/solutions/case-studies/peak/ Web Expedia https://aws.amazon.com/solutions/case-studies/expedia/ Nordstrom https://aws.amazon.com/solutions/case-studies/nordstrom/ JustGiving http://aws.amazon.com/solutions/case-studies/justgiving/ Tokyu Hands https://aws.amazon.com/blogs/aws/how-tokyu-hands-architected-a-cost-effective-shopping-system-with-amazon-dynamodb/ jobandtalent https://aws.amazon.com/solutions/case-studies/jobandtalent/ Tigerspike http://aws.amazon.com/solutions/case-studies/tigerspike/
- #58 Amazon’s path from Relational Databases to NoSQL reflects the journey many customers are now taking. Amazon.com, the online retail business, runs on one of the world’s largest web infrastructures. Back in 2004, Amazon.com was using Relational Oracle Databases and they were unable to scale their relational database. Maintenance and adminstration. In order to keep Amazon.com highly scalable to support all the incoming traffic, Internal project to investigate options… “If availability, durability, and scalability are the priority, what would the database look like?”. This resulted in a whitepaper that described what the database should look like. This paper made the way for many NoSQL technologies out there today. This was also the beginning of DynamoDB. Database as a Swiss Army Knife - Hundreds of applications built on RDBMS, Poor Scalability (Q4 was a pain), Poor availability, Exorbitantly high costs for h/w, software, admin Dynamo = replicated DHT with consistency management Specialist tool with limited query and simpler consistency Problem: required significant effort to maintain DynamoDB was designed to deliver consistently high performance at any scale: Predictable Performance Massively Scalable Fully Managed Low Cost Now consider PrimeDay 2017 to see how far we came. Amazon DynamoDB requests from Alexa, the Amazon.com sites, and the Amazon fulfillment centers totaled 3.34 trillion, peaking at 12.9 million per second. According to the team, the extreme scale, consistent performance, and high availability of DynamoDB met needs of Prime Day without breaking a sweat.
- #59 Amazon DynamoDB supports multiple high-traffic sites and systems including Alexa, the Amazon.com sites, and all 442 Amazon fulfillment centers. Across the 48 hours of Prime Day, these sources made 7.11 trillion calls to the DynamoDB API, peaking at 45.4 million requests per second.
- #63 If you are familiar with tagging AWS resources for keeping track of expenses, this screen will look familiar. Here’s a screenshot of a daily report using the cost allocation tags. You can clearly identity on each day which services cost how much. This feature will allow you to see what you spending for your tables, indexes etc.
- #64 For eg., you can tag DynamoDB tables for different environments using {env=DEV; env=TEST; env=PROD etc. Here name of the tag is ‘env’ and possible values are DEV/TEST/PROD etc. Then you can clearly see your spend for each environment.
- #68 You can download and run DynamoDB on your local machine for Dev/Test. -- No API charges -- No Data transfer charges. Except for the endpoint, applications that run with the downloadable version of DynamoDB will work with the DynamoDB web service. With minor changes to point your applications to production DynamoDB endpoint http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html https://amazon-dynamodb-labs.com/