Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS re:Invent 2016: Amazon Aurora Best Practices: Getting the Best Out of Your Databases (DAT301)


Published on

Amazon Aurora is a fully managed relational database engine that provides higher performance, availability and durability than previously possible using conventional monolithic database architectures. After launching a year ago, we continued adding many new features and capabilities to Aurora. In this session AWS Aurora experts will discuss the best practices that will help you put these capabilities to the best use. You will also hear from Amazon Aurora customer Intercom on the best practices they adopted for moving live databases with over two billion rows to a new datastore in Amazon Aurora with almost no downtime or lost records.

Intercom was founded to provide a fundamentally new way for Internet businesses to communicate with customers at scale. For growing startups like Intercom, it’s natural for the load on datastores to grow on a weekly basis. The usual solution to this problem is to get a bigger box from AWS. But very soon you reach a point where bigger boat is not an option anymore. You will learn about the benefits of moving to such a datastore, the problems it introduced, and all about the new ability for scaling that was not there before.

Published in: Technology
  • Be the first to comment

AWS re:Invent 2016: Amazon Aurora Best Practices: Getting the Best Out of Your Databases (DAT301)

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Puneet Agarwal – AWS Solutions Architect Steve Abraham – AWS Solutions Architect Mario Kostelac – Intercom Product Engineer November 30, 2016 Amazon Aurora Best Practices Getting the Best Out of Your Databases DAT301
  2. 2. What to Expect from the Session • Migration best practices • Performance best practices • Real-time reporting and analytics • Concurrent event stores • Integration with AWS services • Welcome Intercom!
  3. 3. Amazon Aurora • MySQL-compatible relational database • Performance and availability of commercial databases • Simplicity and cost effectiveness of open source databases • Delivered as managed service Fastest growing service in AWS history
  4. 4. Best practices: Migrations
  5. 5. Amazon Aurora migration options Source database From where Recommended option RDS EC2, on-premises EC2, on-premises, RDS Console based automated snapshot ingestion and catch up via binlog replication. Binary snapshot ingestion through S3 and catch up via binlog replication. Schema conversion using SCT and data migration via DMS.
  6. 6. DB snapshot migration  One-click migration from RDS MySQL 5.6 to Aurora  Automatic conversion from MyISAM to InnoDB  Most migrations take <1 hr, longer for large databases  One click replication from RDS MySQL to Amazon Aurora DB Snapshot One-click Migrate RDS MySQL Master/Slave New Aurora Cluster
  7. 7. Migrating from self-managed MySQL to Aurora • Import MySQL snapshots into Aurora through S3 1) Execute Percona XtraBackup 2) Upload database snapshot to S3 3) Import snapshot from S3 to Aurora cluster 4) Setup logical replication to catch-up 5) Transition database workload to Aurora cluster • Faster migration for large databases (1+ TB) • Import schema and data in one operation
  8. 8. Demo: Restore from S3
  9. 9. Best practices – binary snapshot ingestion • File splitting and compression recommended for 1+ TB databases • Supported file compression formats: 1. Gzip (.gz) 2. Percona xbstream (.xbstream) • Sample compression and splitting command: • innobackupex --user=myuser --password=<password> --stream=tar /mydata/s3-restore/backup | gzip | split -d --bytes=512000 - /mydata/s3-restore/backup3/backup.tar.gz • Import separately: 1. User accounts/passwords 2. Functions 3. Stored procedures
  10. 10. Data copy: Existing data is copied from source tables to tables on the target. Chance data capture and apply: Changes to data on source are captured while the tables are loaded. Once load is complete, buffered changes are applied to the target.  Additional changes captured on the source are applied to the target until the task stopped or terminatedAWS Database Migration Service AWS Schema Conversion Tool Oracle, SQL Server to Aurora Migration Assessment report: SCT analyses the source database and provides a report with a recommended target engine and information on automatic and manual conversions Code Browser and recommendation engine: Highlights places that require manual edits and provides architectural and design guidelines.
  11. 11. Demo: Oracle to Aurora migration
  12. 12. Migration best practices • Use the right migration approach for your use case • Test, migrate, test again! • Consolidate shards on Aurora • Schema conversion • Schema optimization post conversion • For tables with wide text columns, enable DYNAMIC row format • Primary key implementation differences
  13. 13. Best practices: Performance
  14. 14. Aurora performance characteristics  Optimized for highly concurrent random access (OLTP)  Performance scales with number of connections  Consistent performance as number of databases/schemas/tables increase  Consistent performance with increasing number of Aurora read-replicas  Low replication lag
  15. 15. Performance testing https ://d0.a wsstat ic . com /product -m ark eting/Aurora /R DS_ Auro ra_Perf orm ance_Assessm ent_Benchm ark ing_v 1-2 .pdf AMAZON AURORA R3.8XLARGE R3.8XLARGE R3.8XLARGE R3.8XLARGE R3.8XLARGE • Create an Amazon VPC (or use an existing one). • Create four EC2 R3.8XL client instances to run the SysBench client. All four should be in the same AZ. • Enable enhanced networking on your clients. • Tune your Linux settings (see whitepaper). • Install Sysbench version 0.5. • Launch a r3.8xlarge Amazon Aurora DB instance in the same VPC and AZ as your clients. • Start your benchmark! 1 2 3 4 5 6 7
  16. 16. Performance Best Practices MySQL/RDBMS practices still apply  Choose the right tool for the right job (OLAP vs OLTP)  Create appropriate indexes  Tune your SQL code, use explain plans, performance schema Leverage high concurrency  Aurora throughput increases with number of connections  Architect your applications to leverage high concurrency in Aurora Read Scaling  Aurora offers read replicas with virtually no replication lag  Leverage multiple read replicas to distribute your reads
  17. 17. Performance Best Practices Parameter tuning  No need to migrate your performance-related MySQL parameters to Aurora  Aurora Parameter Groups are pre-tuned and already optimal in most cases Performance comparison  Don’t obsess over individual metrics (CPU, IOPS, IO throughput)  Focus on what matters, i.e., application performance Other best practices  Keep query cache on  Leverage CloudWatch metrics
  18. 18. Best Practices: Real-Time Reporting & Analytics
  19. 19. Scenario • Travel & Booking Industry • Live Contextual Product Recommendations • Near Real-Time Reporting • ~700+ Users • ~8 TB Dataset • Usage cycles over 24 hour period • Cost Considerations
  20. 20. Database EngineStorage Backend Application Users Challenges – Original Design
  21. 21. Aurora Cluster Application Users DNS Endpoint Load Balanced Solutions – New Design
  22. 22. Scheduled Job Reads Instance Load Metrics and Calls Lambda Cluster Instances Added, Removed or Resized Lambda Function Applies Logic and Calls RDS API Desired Scale Achieved Solutions – Fleet Scaling
  23. 23. Best practices: Massively Concurrent Event Stores
  24. 24. Scenario • Gaming Industry • Millions of RPS • Consistent Latency • Cost Considerations
  25. 25. NoSQL (performance, steady state) Partitioned NoSQL Table User Applications User Applications Optimal Performance Under Moderate Load
  26. 26. NoSQL (performance “hot” state) Partitioned NoSQL Table User Applications User Applications Degraded Performance Under Heavy Load
  27. 27. Aurora implementation (performance) Aurora ClusterUser Applications User Applications Consistent Performance Under Load
  28. 28. NoSQL implementation (cost) Partitioned NoSQL Table User Applications Each Read/Write Billed Separately
  29. 29. Aurora implementation (cost) Aurora ClusterUser Applications Most Operations Served From Memory Small Portion of IO Requests Billed Cost-Efficient Storage
  30. 30. Best practices: AWS Service Integrations
  31. 31. AWS Services Generating Data Amazon S3 Data Lake Amazon Aurora Load From S3 S3 Event Driven Lambda Call Lambda Call S3 Load Completed Notification Delivered Data Flow Call / Notification Flow Event Driven Data Pipeline
  32. 32. User Modifies a Monitored Table Table Trigger Invokes Lambda Lambda Function Applies Logic Security Notification Delivered Event Driven Audit Notification
  33. 33. Demo: Lambda Integration
  34. 34. Amazon SQS Aurora Cluster AWS Lambda Amazon SNSAmazon CloudWatch AWS Lambda Demo: Lambda Integration
  35. 35. Mario Kostelac Product engineer mariokostelac
  36. 36. ?
  37. 37. How did we start using Aurora?
  38. 38. <>
  39. 39. Our stack <whatever>.js
  40. 40. Our first MySQL instance
  41. 41. Then we got bigger...
  42. 42. And bigger…
  43. 43. Eventually, two MySQL instances…
  44. 44. Why so slow?
  45. 45. 2 billion rows problem - Your table can’t fit in RAM - Your table can’t fit in RAM! - You can’t modify your table schema
  46. 46.
  47. 47. Replace RDS MySQL with? - DynamoDB - Partitioned RDS MySQL - Aurora?
  48. 48. Is Aurora good enough for us?
  49. 49. Test your load!
  50. 50. “The only testing that should matter to you is testing against YOUR production load!” me, right now!
  51. 51. How to test your load? 1. Test your tools (why?) 2. Create an image of your load! (how?) 3. Test your load!
  52. 52. MySQL prod Aurora MySQL prod How we migrated from RDS MySQL to Aurora?
  53. 53. Write a migration runbook! 1. Downtimes are stressful 2. Induced downtimes have to be carefully planned
  54. 54. 🚀 Migrated within 8 minutes of downtime, no records lost!
  55. 55. How does it work for us?
  56. 56. Use secondaries when you can
  57. 57. Check your drivers
  58. 58. Build your tooling
  59. 59. What don’t we have to do anymore? Cluster monitoring got simpler Parameter tweaking
  60. 60. So you say it’s impossible to break it?
  61. 61. Test your load! Write a runbook! Use secondaries Check your drivers! Build your tooling
  62. 62. Thank you!
  63. 63. Related Sessions • DAT203 - Getting Started with Amazon Aurora • DAT303 - Deep Dive on Amazon Aurora • DAT302 - Best Practices for Migrating from Commercial Database Engines to Amazon Aurora or PostgreSQL
  64. 64. Remember to complete your evaluations!