Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Service - February 2017 Online Tech Talks

4,192 views

Published on

If you’re running a MySQL database at scale, there’s a good chance you’re sharding your database deployment. Sharding is a useful way to increase the scale of your deployment, but it has drawbacks like higher costs, high administration overheard and lower elasticity. It’s harder to grow or shrink a sharded database deployment to match your traffic patterns. In this session, we will discuss and demonstrate how to use AWS Database Migration Service to consolidate multiple MySQL shards into an Amazon Aurora cluster to reduce cost, improve elasticity and make it easier to manage your database.

Learning Objectives:
Learn how to scale your MySQL database at reduced cost and higher elasticity, by consolidating multiple shards into one Amazon Aurora cluster.

Published in: Technology
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Service - February 2017 Online Tech Talks

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. February 21, 2017 Consolidating MySQL Shards in Aurora AWS Database Migration Service
  2. 2. Agenda • How does the cloud help? • Problem statement • Amazon Aurora overview • Introduction to AWS DMS • Proposed problem solution • Show me!
  3. 3. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cloud benefits
  4. 4. • Multi-engine support: Aurora, MySQL, MariaDB, PostgreSQL, Oracle, SQL Server • Automated provisioning, patching, scaling, backup/restore, failover • High availability with RDS Multi-AZ – 99.95% SLA for Multi-AZ deployments Amazon RDS
  5. 5. • Lower TCO because we manage the muck • Get more leverage from your teams • Focus on the things that differentiate you • Built-in high availability and cross region replication across multiple data centers • Available on all engines, including base/standard editions, not just for enterprise editions • Now even a small startup can leverage multiple data centers to design highly available apps with over 99.95% availability. The Cloud makes things cheaper, easier, better
  6. 6. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How did we end up here?
  7. 7. Things change In the beginning… - The system ran fine but then growth happened To solve the problem you could: - Scale up - Scale out
  8. 8. Sharding
  9. 9. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Aurora
  10. 10. Relational databases were not designed for the cloud Multiple layers of functionality all in a monolithic stack SQL Transactions Caching Logging
  11. 11. Not much has changed in last 20 years Even when you scale it out, you’re still replicating the same stack SQL Transactions Caching Logging SQL Transactions Caching Logging Application SQL Transactions Caching Logging SQL Transactions Caching Logging Application SQL Transactions Caching Logging SQL Transactions Caching Logging Storage Application
  12. 12. Scale-out, distributed, log structured storage Master Replica Replica Replica Availability Zone 1 Shared Storage Volume – Transaction Aware Primary Database Node Read Replica / Secondary Node Read Replica / Secondary Node Read Replica / Secondary Node Availability Zone 2 Availability Zone 3 AWS Region Storage Monitoring Database and Instance Monitoring
  13. 13. Meet Amazon Aurora …… Database reimagined for the cloud  Speed and availability of high-end commercial databases  Simplicity and cost-effectiveness of open source databases  Drop-in compatibility with MySQL  Simple pay as you go pricing Delivered as a managed service
  14. 14. Enterprise-class performance • Provides 5X the throughput of standard MySQL running on the same hardware. • Achieve up to 585,000 reads and 100,000 writes per second • Read replicas with <10ms latency
  15. 15. Aurora Scaling With user connection With number of tables With database size - SYSBENCH With database size - TPCC Connections Amazon Aurora RDS MySQL w/ 30K IOPS 50 40,000 10,000 500 71,000 21,000 5,000 110,000 13,000 Tables Amazon Aurora MySQL I2.8XL local SSD RDS MySQL w/ 30K IOPS (single AZ) 10 60,000 18,000 25,000 100 66,000 19,000 23,000 1,000 64,000 7,000 8,000 10,000 54,000 4,000 5,000 8x U P T O F A S T E R 11x U P T O F A S T E R DB Size Amazon Aurora RDS MySQL w/ 30K IOPS 1GB 107,000 8,400 10GB 107,000 2,400 100GB 101,000 1,500 1TB 26,000 1,200 DB Size Amazon Aurora RDS MySQL w/ 30K IOPS 80GB 12,582 585 800GB 9,406 69 21 U P T O F A S T E R 136x U P T O F A S T E R
  16. 16. Real-life data – gaming workload Aurora vs. RDS MySQL – r3.4XL, MAZ Aurora 3X faster on r3.4xlarge
  17. 17. Cost of ownership: Aurora vs. MySQL MySQL configuration hourly cost Primary r3.8XL Standby r3.8XL Replica r3.8XL Replica R3.8XL Storage 6 TB / 10 K PIOP Storage 6 TB / 10 K PIOP Storage 6 TB / 5 K PIOP Storage 6 TB / 5 K PIOP $1.33/hr $1.33/hr $1.33/hr $1.33/hr $2.42/hr $2.42/hr $2.42/hr Instance cost: $5.32 / hr Storage cost: $8.30 / hr Total cost: $13.62 / hr $2,42/hr
  18. 18. Cost of ownership: Aurora vs. MySQL Aurora configuration hourly cost Instance cost: $4.86 / hr Storage cost: $4.43 / hr Total cost: $9.29 / hr Primary r3.8XL Replica r3.8XL Replica R3.8XL Storage / 6 TB $1.62 / hr $1.62 / hr $1.62 / hr $4.43 / hr *At a macro level Aurora saves over 50% in storage cost compared to RDS MySQL. 31.8% Savings  No idle standby instance  Single shared storage volume  No PIOPs – pay for use I/O  Reduction in overall IOP
  19. 19. Cost of ownership: Aurora vs. MySQL Further opportunity for saving Instance cost: $2.43 / hr Storage cost: $4.43 / hr Total cost: $6.86 / hrStorage IOPs assumptions: 1. Average IOPs is 50% of Max IOPs 2. 50% savings from shipping logs vs. full pages 49.6% Savings Primary r3.8XL Replica r3.8XL Replica r3.8XL Storage / 6TB $0.81 / hr $0.81 / hr $0.81 / hr $4.43 / hr r3.4XL r3.4XL r3.4XL  Use smaller instance size  Pay-as-you-go storage
  20. 20. Use case: MySQL shard consolidation Master Read Replica Shared distributed storage volume M S M M M S S S MySQL shards Aurora cluster Customer, a global SAAS provider, was using hundreds of MySQL shards in order to avoid MySQL performance and connection scalability bottlenecks  Consolidated multiple 29 MySQL shards to single r3.4xlarge Aurora cluster  Even after consolidation cluster utilization is still 30% with plenty of headroom to grow.
  21. 21. Amazon Aurora Customers
  22. 22. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Database Migration Service
  23. 23. AWS Database Migration Service (AWS DMS) DMS migrates databases to AWS easily and securely with minimal downtime. It can migrate your data to and from most widely used commercial and open-source databases – and for as little a $3 for TB DB. Amazon Aurora
  24. 24. Customer premises Application users AWS Internet VPN Start a replication instance Connect to source and target databases Select tables, schemas, or databases  Let AWS DMS create tables, load data, and keep them in sync  Switch applications over to the target at your convenience Keep your apps running during the migration AWS DMS
  25. 25. Load is table by table Replication instance Source Target
  26. 26. Change data capture (CDC) and apply Replication instance Source Target Update t1 t2 t1 t2 Transactions Change apply after bulk load
  27. 27. Replication instance Source Target What else can I do? Source Source
  28. 28. Take it all—or not Source Target Replication instance instance
  29. 29. AWS Database Migration Service Customers
  30. 30. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How can we make it better?
  31. 31. Establish a beach head Before After
  32. 32. Validate
  33. 33. Partial Migration
  34. 34. The Result
  35. 35. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo
  36. 36. Thank you! aws.amazon.com/dms aws.amazon.com/rds/aurora Learn more..

×