Building Scalable Databases on AWS - AWS Summit 2012 - NYC


Published on

Published in: Technology
1 Comment
  • RDS rocks always and it has come upto mature level since 2012 with various offerings.

    I would like to add a nice blog on Scaling with RDS,
    if you are thinking for Horizontal Scaling with RDS , Scalebase is one of the best answers.
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • RDS today is used by thousands of customers from small startups to the world’s largest corporations.We have a predictable history of delivering reliable services and adding even greater functionality over time. (MAZ, RR, Caching)
  • RDS prestructure before adding new attributes, with NoSQL you do not have to do this it is a sparse schema. RDS creates DB on EC2 and EBS for you, gives you Host name and put and you take it from there with your fav SQL client. SimpleDB is in the DataPath, you dump the data and it creates the schema (Does Dynamo?)RDS Benefits – 1. Scale with modify API but there is an outage when you swap out instance types (e.g. from med to XL) but only takes a femins. Events are starting and done. IP address stays the same. Same for EBS add storage with no downtime or degradation in perf2. DB Backup and Restore – DB Snapshot or automated backup. 3. Monitor DB – recover from DB crash, EC2 crash, Gumi has been our number 1 customer for a few weeks over $250k in the last 4 weeksFunzio is in the top 5 and so is activisionEA, sony, bightgames also customers.We have been talking to playfish about DynamoDB but I don’t see them in the rds user base.Could not find zynga.
  • Standard N-Tier Layered Architecture : Logic Tier talks to Data TierData Tier: we have many different DB's depending on the problem space:SimpleDB : Operational Configuration and State goes hereNoSQL : Content Processing (e.g. article categorization), usage processing (e.g. what topics do most readers in England care about at 8am versus 10pm), etc...Memcached (a.k.a ElastiCache) : If you will query it twice based on simple key/value, it goes here.RDS: Reliable AND Complex queries needed? Go hereIn general, we found per user data needed by the presentation tier is best stored using RDS since most requests are user/session focused and requires complex logic based on new features or user interface designs. This includes stuff such as what social networks you belong to, your social interactions such as liking photos or reading articles, synchronization of state across multiple devices, etc.
  • So, who is using DynamoDB?Many companies within Amazon and outside Amazon.Amazon CloudDrive, Smugmug, Amazon retail are examples within Amazon.Smugmug is using us for storing their metadata on DynamoDB.Formpsring and Tapjoy are one of our early adopters.Finally, the shazam the popular voice recognition application for ads and music is using us to store its data for various applications.
  • Building Scalable Databases on AWS - AWS Summit 2012 - NYC

    1. 1. Building Scalable DatabaseApplications on AWSSundar Raghavan, AmazonGreg Scallan, Architect, FlipboardEric Weller, Director, Earth NetworksEdward Dingels, Technical Lead, Earth Networks
    2. 2. Database Services: One Size Does Not Fit All Amazon RDS + DynamoDB ElastiCache MySQL, Oracle, Memcached Apps that need Scalable Relational databases Apps that need Massive Scalability YesSQL NoSQL
    3. 3. Building Database Applications – The Old WayDemand Human Layer • Stuck with peak capacity • Human driven • Time consuming Q1 Time
    4. 4. What We Hear From Customers “Help us focus on applications – Shift database maintenance time to more productive application development time” Security planning Frequent server upgrades License/doc training Backup rec 5% 5% Constant storage upgrades load/unload 20% 5% Scripting coding Backup and recovery Software upgrade and patching Performance and 25% 40% tuning Hardware failure replacements Install, upgrade, patch, migration Distribution of timeSource:,5180,-1110,00.ppt
    5. 5. Building Database Applications – The New Way Application LayerDemand • Elastic capacity • App driven • Immediate Time
    6. 6. Amazon Relational Database ServiceRDS is a fully managed Relational database service that is simple to deploy, easy to scale, reliable and cost-effective Choice of Database Engines Fully Managed Service Push Button Scalability Fault Tolerance with Multi-AZ Works with EC2 & ElastiCache Amazon Relational Database Service (RDS)
    7. 7. High Performance Relational Databases Amazon RDS Improve Increase Reduce Configuration Availability Throughput Latency Push-Button Scaling Multi AZ Read Replicas ElastiCachePush-Button Scaling Read Replicas ElastiCache
    8. 8. Push Button ScalingScale up your instanceand storageScale out as your appgrowsAutomatic backupEasy restore
    9. 9. Read Scaling with Amazon ElastiCache and Read Replicas Improve performance with ElastiCache Read replica for read scaling Off load BI and reporting from production
    10. 10. Reliability For Production Applications
    11. 11. High Availability – Multi-AZ Deployments Enterprise grade fault tolerance Synchronous replication Automatic failover
    12. 12. Amazon RDS Customer Highlights Gaming Web Apps Mobile/Social Media
    13. 13. IntroductionYour Social Magazine for Apple Mobile DevicesDelivers relevant articles and photos based on usage andinteractions within your social networksLaunched 6 months after the initial team put togetherOver 5 million users and 2 billion page “flips” per month
    14. 14. Operating in the Cloud : Managing Complex, Real Time Data 6 months to deploy a real-time, socially relevant magazineChallenge Constantly changing user interestsArchitecture Ability to change all hardware and software elastically Frequently changing system requirementsApp needs Complex queries on user and relevancy data Milliseconds count. So, does uptime Solution Highly performant, reliable, proven database technology Amazon RDS MySQL
    15. 15. The Data View of the World Flipboard Application ElastiCache Memcache for performance Reliable andOperational Complex data,Configuration Queriesand State Data SimpleDB RDS MySQL
    16. 16. Friends, Magazines … anyone relevant to me who published something very recently that I care about seeing Show Me More, Please!A friend who recently shared a photo
    17. 17. Amazon RDS Tips For Success1. Leverage the Cloud for what it does best. Don’t bring old DC habits.2. Scale up for better performance.3. Use RDS for complex, real-time data.4. Use Read Replicas to augment write heavy databases. They are awesome.5. Leverage existing SQL knowledge and experience.6. Use copies of your database for testing new code. It’s trivial and saves time.7. Scale horizontally with sharding. Plan for it before you need it.
    18. 18. SQL or NoSQL?
    19. 19. Relational Or Non-Relational?Factors Relational (RDS) NoSQL (DynamoDB) • Existing database apps • New Web scale applicationsApplication type • Business process-centric apps • Large # of small writes and readsApplication • Relational data models, transactions • Simple data models, transactionscharacteristics • Complex queries, joins and updates • Range queries, simple updatesScaling • Application or DBA architected • Seamless, on-demand scaling • Performance – developer architected • Performance – AutomaticQoS • Reliability and availability • Reliability and availabilitySkillset • SQL + Programming languages • Web style programming Possible to use both SQL and NoSQL models in one application
    20. 20. The “Big Data” Scalability Challenge Requirement: predictable, consistent performancePerformance  Hardware purchase and provisioning $!   Data sharding Data caching  Cluster management Reality: performance  Fault management degrades with scale Scalability
    21. 21. Kit, go fasterTransformation 3: From Scaling by Architecture … to Scaling By Yes Command Michael
    22. 22. Amazon DynamoDBDynamoDB is a fully managed NoSQL database service that providesextremely fast and predictable performance with seamless scalability Easy Administration Low Latency SSD’s Reserved Capacity Unlimited Potential Storage and Throughput
    23. 23. Amazon DynamoDB NoSQL Database Fast & predictable performance Seamless Scalability ADMIN Easy administration“Even though we have years of experience with large, complexNoSQL architectures, we are happy to be finally out of thebusiness of managing it ourselves.” - Don MacAskill, CEO
    24. 24. DynamoDB Highlights Low Latency • SSD-based storage nodes • Typical latency =Get latency single-digit millisecondsCPU utilization
    25. 25. Provisioned Throughput Reserve the IOPS needed for each table Set at table creation Increase / decrease any time via API call Pay for throughput and storage (not instances) • $0.01 per hour for every 10 units of Write Capacity • $0.01 per hour for every 50 units of Read Capacity • $1.00 per GB-month of Storage
    26. 26. DynamoDB Customer Highlights
    27. 27. Earth Networks Case StudyEric Weller – Director of DevelopmentEdward Dingels – Technical Lead
    28. 28. IntroductionGathers and analyzes atmospheric observations from aglobal sensor network to promote a better understandingof the planetProprietary lightning network output used to pinpointlightning activity - best indicator of dangerous weatherOwner of the WeatherBug brand (mobile, desktop, Web)
    29. 29. Problem Generate lightning alert notifications, in proximity Need to the user’s location, on a mobile device. Geospatial queries ScalableConstraints • 6 million existing mobile users • 100% YOY mobile growth • Severe Weather Outbreaks Fast • Speed + Accuracy = Safety Reduce Time to Market Cost of Ownership
    30. 30. Analysis Provider Product Throughput Engineering Cost of per Instance Cost OwnershipMicrosoft SQL Server 2008 Medium Low HighMySQL MySQL Medium High* High*Earth Networks In Memory Quadtree High High MediumAmazon Mem-Cache High Medium MediumAmazon DynamoDB High Low Low * Not currently supported by Earth Networks
    31. 31. SolutionGeo-Hash lightning dataBackend Windows service for loading/managementof data inside DynamoDBFront-end web service for encoding/decodingUse geometric instead of trigonometric calculationsClient computes actual range and produces alerts
    32. 32. ReflectionDynamoDB Advantages• Easy to Provision• Built-in Consistency• Scalable• High AvailabilityDynamoDB Wish List• Automated regional DR failover• Durability across regions in addition to inside a region• Auto-scale down without a throttle on change• Item size limit of 64k
    33. 33. Lightning Demo
    34. 34. Try Amazon DB Services!Amazon RDS – Free usage• For Details Visit – Free usage• For Details Visit