Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

What's New in Amazon Aurora (DAT204-R1) - AWS re:Invent 2018

1,494 views

Published on

Amazon Aurora is a MySQL- and PostgreSQL-compatible relational database with the speed, reliability, and availability of commercial databases at one-tenth the cost. This session provides an overview of Aurora, explores recently announced features, such as Serverless, Multi-Master, and Performance Insights, and helps you get started.

  • Be the first to comment

What's New in Amazon Aurora (DAT204-R1) - AWS re:Invent 2018

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What’s New in Amazon Aurora Debanjan Saha GM, Amazon Aurora & RDS Amazon Web Services D A T 2 0 4
  2. 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Relational Database Service (Amazon RDS) Choice Value Innovation
  3. 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon RDS Choice of open source and commercial databases RDS Platform Open Source Engines Commercial Engines Advanced monitoring Routine maintenance Push-button scaling Automatic fail-over Backup & recovery X-region replication Isolation & security Industry compliance Automated patching Cloud Native Engine
  4. 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora… Enterprise database at open source price Delivered as a managed service Amazon Aurora Speed and availability of high-end commercial databases Simplicity and cost-effectiveness of open source databases Drop-in compatibility with MySQL and PostgreSQL Simple pay as you go pricing
  5. 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora innovations Re-imagining databases for the cloud Automate administrative tasks – fully managed service Scale-out, distributed, multi-tenant design Service-oriented architecture leveraging AWS services
  6. 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scale-out, distributed architecture Purpose-built log-structured distributed storage system designed for databases Storage volume is striped across hundreds of storage nodes distributed over 3 different availability zones Six copies of data, two copies in each availability zone to protect against AZ+1 failures Plan to apply same principles to other layers of the stack Shared storage volume Storage nodes with SSDs Availability Zone 1 SQL Transactions Caching Availability Zone 2 SQL Transactions Caching Availability Zone 3 SQL Transactions Caching
  7. 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Leveraging AWS services Invoke AWS Lambda events from stored procedures/triggers Load data from Amazon Simple Storage Service (Amazon S3), store snapshots and backups in S3 Lambda function Amazon S3 AWS Identity and Access Management Amazon CloudWatch Use AWS Identity and Access Management (IAM) roles to manage database access control Upload systems metrics and audit logs to CloudWatch
  8. 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Automate administrative tasks Automatic fail-over Backup & recovery Isolation & security Industry compliance Push-button scaling Automated patching Advanced monitoring Routine maintenance Takes care of your time-consuming database management tasks, freeing you to focus on your applications and business You AWS Schema design Query construction Query optimization
  9. 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora customer adoption Fastest growing service in AWS history Aurora is used by ¾ of the top 100 AWS customers
  10. 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Who are moving to Aurora and why? Customers using open source engines • Higher performance – up to 5x • Better availability and durability • Reduces cost – up to 60% • Easy migration; no application change Customers using commercial engines • One tenth of the cost; no licenses • Integration with cloud ecosystem • Comparable performance and availability • Migration tooling and services
  11. 11. Aurora performance 5x faster than MySQL; 3x faster than PostgreSQL
  12. 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Write and read throughput Aurora MySQL is 5x faster than MySQL 0 50,000 100,000 150,000 200,000 250,000 Series1 Series2 Series3 Series4 Series5 0 100,000 200,000 300,000 400,000 500,000 600,000 700,000 800,000 Series1 Series2 Series3 Series4 Series5 Write Throughput Read Throughput Using Sysbench with 250 tables and 200,000 rows per table on R4.16XL
  13. 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Series1 Series1 Series2 Series2 Series3 Series3 0 500 1000 1500 2000 2500 3000 3500 4000 1 2 Runtime (seconds) pgbench initialization, scale 10000 (150 GiB) 86% reduction in vacuum time Bulk data load performance Aurora PostgreSQL loads data 2x faster than PostgreSQL
  14. 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Bulk data load performance Aurora MySQL loads data 2.5x faster than MySQL Series1 Series1 Series2 Series2 0 100 200 300 400 500 600 700 800 1 2 Runtime (sec.) 10 Sysbench Tables, 10MM rows per each
  15. 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 0.00 100.00 200.00 300.00 400.00 500.00 600.00 700.00 1 61 121 181 241 301 361 421 481 541 601 661 721 781 841 901 961 1,021 1,081 1,141 1,201 Responsetime,ms SYSBENCH RESPONSE TIME (p95), 30 GiB, 1024 CLIENTS Series1 Series2 Performance variability under load Aurora PostgreSQL is ~10x more consistent than PostgreSQL
  16. 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Sysbench write-only workload with 250 tables and 200,000 initial rows per table 0 500 1,000 1,500 2,000 2,500 0 100 200 300 400 500 600 Time from start of run (sec.) Write Response Time (ms.) Series1 Series2 Performance variability under load Aurora MySQL is ~25x more consistent than MySQL
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 0 50 100 150 200 250 1 2 3 4 Max write throughput – up 100% 0 100 200 300 400 500 600 700 800 1 2 3 4 Max read throughput – up 42% Launched with R3.8xl 32 cores, 256GB memory Now support R4.16xl 64 cores, 512GB memory R5.24xl coming soon 96 cores, 768GB memory Besides many performance optimizations, we are also upgrading HW platform Performance improvement over time Aurora MySQL – 2015-2018
  18. 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How did we achieve this? Do less work • Do fewer IOs • Minimize network packets • Cache prior results • Offload the database engine Be more efficient • Process asynchronously • Reduce latency path • Use lock-free data structures • Batch operations together • Databases are all about I/O • Network-attached storage is all about packets/second • High-throughput processing is all about context switches
  19. 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora I/O profile MySQL with Replica Amazon Aurora EBS mirrorEBS mirror AZ 1 AZ 2 EBS Amazon Elastic Block Store (EBS) Primary Instance Replica Instance 1 2 3 4 5 Amazon S3 MySQL I/O profile for 30 min Sysbench run 780K transactions 7,388K I/Os per million txns (excludes mirroring, standby) Average 7.4 I/Os per transaction AZ 1 AZ 3 Primary Instance AZ 2 Replica Instance ASYNC 4/6 QUORUM Distributed writes Replica Instance Amazon S3 Aurora IO profile for 30 min Sysbench run 27,378K transactions – 35X MORE 0.95 I/Os per transaction (6X amplification) – 7.7X LESS Binlog Data Double-writeLog From files
  20. 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora lock management MySQL lock manager Aurora lock manager Scan Delete Insert Scan Scan Delete Scan Insert Insert Delete Insert Scan Same locking semantics as MySQL Concurrent access to lock chains Multiple scanners in individual lock chains Lock-free deadlock detection Needed to support many concurrent sessions, high update throughput
  21. 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous key prefetch  Works for queries using Batched Key Access (BKA) join algorithm + Multi-Range Read (MRR) Optimization  Performs a secondary to primary index lookup during JOIN evaluation  Uses background thread to asynchronously load pages into memory Latency improvement factor vs. Batched Key Access (BKA) join algorithm. Decision support benchmark, R3.8xlarge 14.57 - 2.00 4.00 6.00 8.00 10.00 12.00 14.00 16.00 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Series1 AKP used in queries 2, 5, 8, 9, 11, 13, 17, 18, 19, 20, 21
  22. 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Batched scans Standard MySQL pprocesses rows one-at- a-time resulting in high overhead due to • Repeated function calls • Locking and latching • Cursor store and restore • InnoDB to MySQL format conversion Amazon Aurora scan tuples from the InnoDB buffer pool in batches for • Table full scans • Index full scans • Index range scans Latency improvement factor vs. Batched Key Access (BKA) join algorithm. Decision support benchmark, R3.8xlarge 1.78X - 0.20 0.40 0.60 0.80 1.00 1.20 1.40 1.60 1.80 2.00 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
  23. 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Parallel query processing Aurora storage has thousands of CPUs • Opportunity to push down and parallelize query processing • Moving processing close to data reduces network traffic and latency However, there are significant challenges • Data is not range partitioned – require full scans • Data may be in-flight • Read views may not allow viewing most recent data • Not all functions can be pushed down Database Node Storage nodes Push down predicates Aggregate results
  24. 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Well-known decision support benchmark 0x 20x 40x 60x 80x 100x 120x 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Query response time reduction  Peak speed up ~120x  >10x speedup: 8 of 22 queries We were able to test Aurora’s parallel query feature and the performance gains were very good. To be specific, We were able to reduce the instance type from r3.8xlarge to r3.2xlarge. For this use-case, parallel query was a great win for us. Jyoti Shandil, Cloud Data Architect Performance results
  25. 25. What about availability “Performance only matters if your database is up”
  26. 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 6-way replicated storage Survives catastrophic failures • Six copies across three availability zones • 4 out 6 write quorum; 3 out of 6 read quorum • Peer-to-peer replication for repairs • Volume striped across hundreds of storage nodes SQL Transaction AZ 1 AZ 2 AZ 3 Caching SQL Transaction AZ 1 AZ 2 AZ 3 Caching
  27. 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Read replica and custom end-point Master Read replica Read replica Read replica Shared distributed storage volume Reader end-point #1 Reader end-point #2 Up to 15 promotable read replicas across multiple availability zones Re-do log based replication leads to low replica lag – typically < 10ms Custom reader end-point with configurable failover order
  28. 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Instant crash recovery Traditional database Have to replay logs since the last checkpoint Typically 5 minutes between checkpoints Single-threaded in MySQL; requires a large number of disk accesses Amazon Aurora Underlying storage replays redo records on demand as part of a disk read Parallel, distributed, asynchronous No replay for startup Checkpointed Data Redo Log Crash at T0 requires a re-application of the SQL in the redo log since last checkpoint T0 T0 Crash at T0 will result in redo logs being applied to each segment on demand, in parallel, asynchronously
  29. 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Continuous availability with multi-master Master Master Master Master Shared distributed storage volume Application #1 Application #2 Up to 15 promotable read replicas across multiple availability zones Re-do log based replication leads to low replica lag – typically < 10ms Custom reader end-point with configurable failover order
  30. 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Scaling write workload Optimistic conflict management There are many “oases” of consistency in Aurora Database nodes know transaction orders from that node Storage nodes know transactions orders applied at that node Only have conflicts when data changed at both multiple database nodes AND multiple storage nodes Much less coordination required Near linear throughput scaling for workloads with no or low conflict Lower commit latency for workloads with low conflict
  31. 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Transactions with no conflict  Transactions T1 and T2 from BLUE and ORANGE masters update different tables (pages) Table 1 and Table 2.  No logical or physical conflicts – no coordination required  Near linear scaling with number of masters for this type of sharded or partitioned workloads. 1 1 1 1 11 2 2 2 2 22 PAGE1 PAGE2 MASTER MASTER Begin Trx (T1) Update (Table1) Commit (T1) Begin Trx (T2) Update (Table2) Commit (T2) TABLE 1 TABLE 2
  32. 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Physical conflict resolution Arbitration at storage node 1 1 1 2 21 PAGE1 PAGE2 MASTER MASTER Begin Trx (T1) Update (Table1) Commit (T1) Begin Trx (T2) Update (Table1) Rollback (T2) TABLE 1 TABLE 2  Transactions T1 and T2 from BLUE and ORANGE masters update the same table (page) Table 1.  Transaction T1 from BLUE master achieves quorum – transaction T1 is committed.  Transaction T2 from ORANGE master fails to achieves quorum – transaction T2 is rolled back.  Storage nodes act as arbitrator for conflict resolution X
  33. 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Logical conflict resolution Arbitration by designated tiebreaker 1 1 1 2 21 1 2 2 2 21 PAGE1 PAGE2 MASTER MASTER Begin Trx (T1) Update (Table1) Update (Table 2) Commit (T1) Begin Trx (T2) Update (Table2) Update (Table1) Rollback (T2) TABLE 1 TABLE 2  Transactions T1 and T2 from BLUE and ORANGE masters update both tables (pages) Table 1 (page 1)and Table 2 (page 2).  Transaction T1 from BLUE master achieves quorum on page 1. Transaction T2 from ORANGE master achieves quorum on page 2.  Designated tiebreaker node resolves conflict in favor of BLUE master – T2 is rolled back; T1 is committed. TIE BREAKER X
  34. 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora Multi-Master – scaling and availability 0 10000 20000 30000 40000 50000 60000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99 101 103 105 107 109 111 113 115 117 119 121 123 125 AggregatedThroughput Time in minutes Sysbench workload on 4 R4.XL nodes Adding a node Adding a node Node going down Node recovering
  35. 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Database backtrack t0 t1 t2 t0 t1 t2 t3 t4 t3 t4 Rewind to t1 Rewind to t3 Invisible Invisible Backtrack brings the database to a point in time without requiring restore from backups • Backtracking from an unintentional DML or DDL operation • Backtrack is not destructive. You can backtrack multiple times to find the right point in time • Also useful for QA (rewind your DB between test runs)
  36. 36. Amazon Aurora is easy to use Automated storage management, security and compliance, advanced monitoring, database migration
  37. 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Performance Insights Dashboard showing database load  Easy – e.g. drag and drop  Powerful – drill down using zoom in Identifies source of bottlenecks  Sort by top SQL  Slice by host, user, wait events Adjustable time frame  Hour, day, week , month  Up to 2 years of data; 7 days free Max vCPU CPU bottleneck SQL w/ high CPU
  38. 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Alexa + Performance Insights Integration between Alexa & Performance Insights built by AWS partner Slalom Uses the Performance Insights API to identify bottlenecks in Amazon RDS Get actionable suggestions such as on-demand scaling, DBA notifications, and paging Go to the Slalom booth (#1438) to see a live demo and learn more
  39. 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Alexa + Performance Insights Demo
  40. 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Security management Encryption to secure data at rest using customer managed keys • AES-256; hardware accelerated • All blocks on disk and in Amazon S3 are encrypted • Key management via AWS KMS Encrypted cross-region replication, snapshot copy—SSL to secure data in transit Advanced auditing and logging without any performance impact Database activity monitoring Database Engine *NEW* Customer Master Key(s) Data key 1 Storage node Storage node Storage node Storage node Data key 1 Data key 1 Data key 1
  41. 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Industry certifications Amazon Aurora gives each database instance IP firewall protection Aurora offers transparent encryption at rest and SSL protection for data in transit Amazon VPC lets you isolate and control network configuration and connect securely to your IT infrastructure AWS Identity and Access Management provides resource- level permission controls *New* *New* *New*
  42. 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CloudWatch consumer Database activity monitoring Search: Look for specific events across log files Metrics: Measure activity in your Aurora DB cluster • Continuously monitor activity in your DB clusters by sending these audit logs to CloudWatch logs • Export to S3 for long term archival; analyze logs using Amazon Athena; visualize logs with Amazon QuickSight Visualizations: Create activity dashboards Alarms: Get notified or take actions Amazon Aurora Amazon CloudWatch 3rd party consumer Amazon Kinesis
  43. 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora migration options Source database From where Recommended option RDS EC2, on premises EC2, on premises, RDS Console based automated snapshot ingestion and catch up via binlog replication. Binary snapshot ingestion through S3 and catch up via binlog replication. Schema conversion using SCT and data migration via DMS.
  44. 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What’s new in AWS DMS and AWS SCT? Workload qualification Allows customers to estimate and manage the efforts required to migrate Oracle and SQL Server workloads to Aurora Migration playbook Step-by-step instructions on how to migrate from Oracle and SQL Server to Aurora Schema conversion Automation from Oracle and SQL Server to Aurora is over 90% Native start point Customers can use engine-native utilities to copy data, such as PG dump and restore, and replicate the changes using DMS
  45. 45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora Serverless use cases Infrequently used applications (e.g., low-volume blog site) Applications with variable load— peaks of activity that are hard to predict (e.g., news site) Development or test databases not needed on nights or weekends Consolidated fleets of multi-tenant SaaS applications
  46. 46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora Serverless Starts up on demand, shuts down when not in use Scales up/down automatically No application impact when scaling Pay per second, 1 minute minimum WARM POOL OF INSTANCES APPLICATION DATABASE STORAGE SCALABLE DB CAPACITY REQUEST ROUTERS
  47. 47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scale up and down with load 1 2 4 8 16 32 64 128 0 500 1000 1500 2000 2500 3000 1 12 23 34 45 56 67 78 89 100 111 122 133 144 155 166 177 188 199 210 221 232 243 254 265 276 287 298 309 320 331 342 353 364 375 386 397 408 419 430 441 452 463 474 485 496 507 518 529 540 551 562 573 584 595 606 617 628 639 650 661 672 683 694 705 716 727 TPS ACU
  48. 48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What’s new in Aurora Serverless New regions Seoul, Singapore, Sydney, Mumbai, London, N. California, Paris, Frankfurt, Canada Central Compliance FedRAMP, HIPPA, PCI, SOC, ISO & HITRUST Preview Support for Aurora PostgreSQL Preview Support for REST DATA API
  49. 49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon RDS Data API for serverless applications Millions of IOT/mobile devices Data API fleet API End-point Amazon Aurora Serverless Access through simple web interface • Public endpoint addressable from anywhere • No client configuration required • No persistent connections required Ideal for Serverless applications (Lambda) Ideal for light-weight applications (IOT)
  50. 50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Related breakouts Tuesday, November 27 DAT305—Deep Dive on Amazon Aurora with PostgreSQL Compatibility 5:30 PM–6:30 PM | Venetian, Level 3, San Polo 3405 DAT304—Deep Dive on Amazon Aurora with MySQL Compatibility 6:15 PM–7:15 PM | Venetian, Level 4, Marcello 4505 Wednesday, November 28 DAT207—Migrating Databases to the Cloud with AWS Database Migration Service 2:30 PM–3:30 PM | Venetian, Level 2, Titian 2202–T1
  51. 51. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Debanjan Saha deban@amazon.com
  52. 52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×