Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Leadership Session: AWS Database and Analytics (DAT206-L) - AWS re:Invent 2018

1,060 views

Published on

We’re witnessing an unprecedented growth in the amount of data collected and stored in the cloud. Getting insights from this data requires database and analytics services that scale and perform in ways not possible before. AWS offers the broadest set of database and analytics services to process, store, manage, and analyze all your data. In this session, we provide an overview of the database and analytics services at AWS, new services and features we launched this year, how customers are using these services, and our vision for continued innovation in this space.

Leadership Session: AWS Database and Analytics (DAT206-L) - AWS re:Invent 2018

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Leadership Session: AWS Databases and Analytics Raju Gulabani Vice President Databases, Analytics, Machine Learning, & Blockchain AWS D A T 2 0 6 - L
  2. 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What to expect Hear about the new announcements, with an emphasis on news not covered in Andy’s keynote 1 Understand our database and analytics strategy; our portfolio of various services & how they work together 2 Plan how you would use these services by appreciating how others use them 3
  3. 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Our strategy & our beliefs 1. There is going to be an explosion in data. 2. Cloud will enable a different architecture. 3. One size does not fit all—databases should be purpose-built. 2010
  4. 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Analytics Our portfolio Broad and deep portfolio, purpose-built for builders Redshift Data warehousing EMR Hadoop + Spark Athena Interactive analytics Kinesis Data Analytics Real time Elasticsearch Service Operational Analytics QuickSight SageMaker S3/Glacier Glue ETL & Data Catalog Lake Formation Data Lakes Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams Data Movement Business Intelligence & Machine Learning Data Lake
  5. 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Analytics Our portfolio Broad and deep portfolio, purpose-built for builders QuickSight SageMaker S3/Glacier Glue ETL & Data Catalog Lake Formation Data Lakes Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams Data Movement Business Intelligence & Machine Learning Data Lake Redshift Data warehousing EMR Hadoop + Spark Kinesis Data Analytics Real time Elasticsearch Service Operational Analytics Athena Interactive analytics RDS MySQL, PostgreSQL, MariaDB, Oracle, SQL Server Aurora MySQL, PostgreSQL DynamoDB Key value, Document ElastiCache Redis, Memcached Neptune Graph Timestream Time Series QLDB Ledger Database RDS on VMware Databases
  6. 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Our portfolio Broad and deep portfolio, purpose-built for builders Redshift Data warehousing EMR Hadoop + Spark Athena Interactive analytics Kinesis Data Analytics Real time Elasticsearch Service Operational Analytics RDS MySQL, PostgreSQL, MariaDB, Oracle, SQL Server Aurora MySQL, PostgreSQL QuickSight SageMaker DynamoDB Key value, Document ElastiCache Redis, Memcached Neptune Graph Timestream Time Series QLDB Ledger Database S3/Glacier Glue ETL & Data Catalog Lake Formation Data Lakes Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams Data Movement Analytics Databases Business Intelligence & Machine Learning Data Lake Managed Blockchain Blockchain Templates Blockchain RDS on VMware
  7. 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Three type of projects Quickly build new apps in the cloud Gain new insights “Lift and shift” existing apps to the cloud
  8. 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Three type of projects Quickly build new apps in the cloud Gain new insights “Lift and shift” existing apps to the cloud
  9. 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Traditionally, analytics looked like this Relational data GBs-TBs scale [not designed for PB/EBs] Expensive: Large initial capex + $10K-$50K/TB/year 90% of data was thrown away because of cost OLTP ERP CRM LOB Data Warehouse Business Intelligence
  10. 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Our beliefs 1. All data has value. No data should be thrown away. 2. All employees should have access to all data (subject to company access rules). 2010
  11. 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Snowball Snowmobile Kinesis Data Firehose Kinesis Data Streams S3 Redshift EMR Athena Kinesis Elasticsearch Service Data lakes on AWS Kinesis Video Streams AI Services QuickSight Exabyte scale Store and analyze relational and non-relational data Purpose-built analytics tools Cost effective • Store at 2.3 cents per GB-month in Amazon S3 • Query with Amazon Athena at ½ cent per GB scanned • DW with Amazon Redshift for $1,000/TB/year Give access to everyone • Amazon QuickSight: $0.30 for 30 minutes of use
  12. 12. CHALLENGE Need to create constant feedback loop for designers. Gain up-to-the-minute understanding of gamer satisfaction to guarantee gamers are engaged, resulting in the most popular game played in the world. Fortnite | 125+ million players
  13. 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Epic Games uses data lakes and analytics Entire analytics platform running on AWS Amazon S3 leveraged as a data lake All telemetry data is collected with Amazon Kinesis Real-time analytics done through Spark on Amazon EMR, DynamoDB to create scoreboards and real-time queries Use Amazon EMR for large batch data processing Game designers use data to inform their decisions Game clients Game servers Launcher Game services N E A R R E A L T I M E P I P E L I N E N E A R R E A L T I M E P I P E L I N E Grafana Scoreboards API Limited raw data (real time ad-hoc SQL) User ETL (metric definition) Spark on EMR DynamoDB NEAR REAL-TIME PIPELINES BATCH PIPELINES ETL using EMR Tableau/BI Ad-hoc SQLS3 (Data lake) Kinesis APIs Databases S3 Other sources
  14. 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Lake Formation (sign up for the preview) Build a secure data lake in days Move, store, catalog, and clean your data faster Move, store, catalog, and clean your data faster with machine learning Enforce security policies across multiple services Enforce security policies across multiple services Gain and manage new insights Empower analyst and data scientist to gain and manage new insights
  15. 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How it works Data lakes and analytics on AWS S3 IAM KMS OLTP ERP CRM LOB Devices Web Sensors Social Kinesis Build data lakes quickly • Identify, crawl, and catalog sources • Ingest and clean data • Transform into optimal formats Simplify security management • Enforce encryption • Define access policies • Implement audit login Enable self-service and combined analytics • Analysts discover all data available for analysis from a single data catalog • Use multiple analytics tools over the same data Athena Redshift AI Services EMR QuickSight Data catalog
  16. 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Redshift Highly scalable cloud data warehouse at 10x the performance and 1/10th the cost of traditional data warehouses Virtually unlimited concurrency Extends your data lake Dynamically scales to support virtually unlimited number of concurrent users and growing data volumes Analyze exabytes of data in the Amazon S3 data lake together with petabytes of data loaded into Amazon Redshift’s high performance SSDs 10x performance 1/10th the cost Get faster time-to-insight for all types of analytics workloads; powered by machine learning, columnar storage and MPP Start at $0.25 per hour, scale out as low as $1,000 per terabyte per year
  17. 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Automatically spins up additional clusters on-demand Handles virtually unlimited number of concurrent users Accrued minutes make it free for most customers Amazon Redshift Concurrency Scaling (Preview) Consistently fast performance at virtually unlimited concurrency Redshift Managed S3 Cluster Leader Node Data Data Caching Layer Cluster Leader Node
  18. 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Redshift Spectrum Extend the data warehouse to exabytes of data in S3 data lake S3 data lakeRedshift data Amazon Redshift Spectrum query engine Exabyte Redshift SQL queries against Amazon S3 Join data across Amazon Redshift and Amazon S3
  19. 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Redshift Elastic Resize (GA) Quickly scale up or down to increase performance on-demand Cluster Leader Node Data Data Data Data Redshift Managed S3 Add/remove additional nodes to cluster in minutes Available today
  20. 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Redshift is >3x faster than 6 months ago 100% 181% 237% 284% 350% Redshift 6 months ago Redshift July 2018 Redshift Aug 2018 Redshift Sep 2018 Redshift Oct 2018 Queriesperhour asa%ofRedshift6monthsago Normalized Queries Per Hour (QPH) (assuming Redshift’s QPH 6 months ago = 100%; higher is better)
  21. 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. “20 percent of our queries now complete in less than one second. Best of all, we didn’t have to change anything to get this speed-up with Redshift, which supports our mission-critical workloads.” -Greg Rokita, Executive Director of Technology, Edmunds
  22. 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fastest: up to 16x faster 100% 34% 6% As a % of Amazon Redshift’s queries per hour Based on the cloud DW benchmark derived from TPC-DS 3 TB dataset, 4-node cluster Queries per hour
  23. 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Most cost effective: 1/10 the cost $ per hour (As a % of Amazon Redshift’s $ per hour with a 3yr RI) Based on the cloud DW benchmark derived from TPC-DS 3 TB dataset, 4-node cluster 2683% 500% 320% 211% 100%
  24. 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CHALLENGE Needed to analyze data to find insights, identify opportunities, and evaluate business performance. The Oracle DW did not scale, was difficult to maintain, and costly. SOLUTION Deployed a data lake with Amazon S3, and run analytics with Amazon Redshift, Amazon Redshift Spectrum, and Amazon EMR. Result: They doubled the data stored (100PB), lowered costs, and was able to gain insights faster. 50 PB of data 600,000 analytics jobs/day
  25. 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Equinox Fitness migrated from Teradata to Redshift Maximilian (ELT scripts) Spark on EMR Redshift S3 Clickstream Cycling logs Club management software Applications Social Redshift Spectrum EMR Athena Equinox apps 3rd party apps Migrated from Teradata data warehouse Built a DW with Redshift and data lake with S3 Analytics on data lake with Amazon Athena, Amazon Redshift Spectrum, and Amazon EMR Increased user productivity to move faster Amazon Redshift costs ~20% of its original Teradata maintenance & support
  26. 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon QuickSight First BI service with pay-per-session pricing for everyone in your organization Serverless, cloud-powered BI service (no servers to manage) Scale from 10s of users to 100s of thousands of users Pay only for what you use • Readers: $0.30/30 min session with a $5/user/month max • Authors: $18/month/Author Integrates with S3, Athena, Redshift, RDS, Aurora, & EMR
  27. 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Directory Service Microsoft AD Custom Date Format Dashboard Save As Aggregate Calculations Readers Groups Private VPC 25 GB SPICE tables Spark and Presto Connector Scheduled refresh Just In Time Provisioning One-click upgrade Search Totals Excel Custom Range 100+ new features released since launch Federated SSO Athena connector Export to CSV S3 Analytics Week Aggregation Aurora PostgreSQL Calculations in SPICE Cross Account S3 Access Aggregate Filters Hourly refresh Row level security Hourly refresh 10K Filter Values On-screen controls Redshift Spectrum Support KPI Chart Spark Connector AWS Directory Service AD Connector Tabular Reports Data labels URL Actions Combo Charts Audit logging with CloudTrail Geospatial maps Count Distinct Parameters Relative Date Filters Filter Groups Table calculations Snowflake Connector SaaS Connectors Teradata Connector HIPAA PCI compliance Amazon QuickSight has been innovating quickly
  28. 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon QuickSight—embedded dashboards Supercharge your applications with embedded dashboards Fully interactive with drill down, filtering, & external links No servers to manage, no long-term commitments Pay for usage with pay-per-session reader pricing Easy embedding with JavaScript SDK
  29. 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Embedded NFL Next Gen Stats Dashboards “With the Amazon QuickSight Readers and pay-per-session pricing, we are able to extend these secure, customized and easy to use dashboards for each club without having to provision servers or manage infrastructure – all while only paying for actual usage.” Matt Swensson Vice President, Emerging Products and Technology Real-time stats for NFL games Embedded in NFL Next Gen Stats Portal Shared with 100s of users across NFL, 32 clubs and broadcast partners
  30. 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon QuickSight is used by customers at the largest scale One of the world’s largest metals and mining companies deployed Amazon QuickSight with its critical risk management (CRM) solution to ensure employee safety. Thousands of employees use its CRM globally. Uses Amazon QuickSight embedded in its Converge Platform, a governance, risk, and compliance healthcare solution. Tens of thousands of users across 900 healthcare organizations use this platform. Amazon.com is using Amazon QuickSight company-wide
  31. 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon QuickSight—ML Insights (Preview) Automated business insights powered by ML and natural language ML-powered anomaly detection ML-powered forecasting Auto-narratives
  32. 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Discover all the hidden trends and anomalies on millions of metrics Amazon QuickSight—ML Insights Example: anomaly detection
  33. 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Sales for office supplies in APAC was 15% above expected.” Amazon QuickSight—ML Insights Example: anomaly detection
  34. 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. “SMB Segment was the top contributor.” Amazon QuickSight—ML Insights Example: anomaly detection
  35. 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. “It’s significant because SMB typically only accounts for 30% of sales.” Amazon QuickSight—ML Insights Example: anomaly detection
  36. 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. QuickSight ML-powered forecasting Traditional BI forecasting Captures seasonality and upward trends Automatically excludes bad data High confidence band Captures only seasonality Missing upward trend Confidence band influenced by bad data QuickSight ML Insights vs. traditional BI forecasting VS.
  37. 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Insights in plain language narrative Embedded within your dashboard No more staring at dashboards for hours! Fully customizable to meet every need No coding needed. Easy-to-use UI templates. Amazon QuickSight—ML Insights Auto-narratives
  38. 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS analytics Any analytic workload, any scale, at the lowest possible cost Redshift Data warehousing EMR Hadoop + Spark Athena Interactive analytics Kinesis Data Analytics Real time Elasticsearch Service Operational Analytics Analytics QuickSight Business Intelligence S3/Glacier Glue ETL & Data Catalog Lake Formation Data Lakes Data Lake
  39. 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. More places to learn about analytics services Amazon Athena Amazon Redshift Amazon Elasticsearch Service AWS Lake Formation ANT401-R2: Deep Dive and Best Practices for Amazon Redshift | Fri 11:30 ANT 401-R1: Deep Dive and Best Practices for Amazon Redshift | Thu 4:00 ANT202-R1: Modern Cloud Data Warehousing ft. Intuit | Thu 2:30 ANT350-R1: What's New with Amazon Redshift ft. McDonald's | Thu 3:15 Sessions that already occured: ANT202-R, ANT350-R ANT323-R1: Build Your Own Log Analytics Solutions on AWS | Thur 11:30 Sessions that already occured: ANT334-R, ANT334-R1, ANT323-RANT203 Introduction to AWS Lake Formation - Build a Secure Data Lake in Days | Wed 7:00pm | Venetian, Level 4, Delfino 4005. Sessions that already occurred: ANT205 ANT340-R1: A Deep Dive into What's New with Amazon EMR | Fri 3:00 Sessions that already occurred: ANT204, ANT312, ANT340-R Amazon Kinesis Amazon QuickSight AWS Glue Introducing Amazon Kinesis Data Analytics for Java applications | Thu 12:15pm | MGM, Level 1, South Concourse 105 ANT322-R1: High Performance Data Streaming with Amazon Kinesis: Best Practices | Thu 1:00 ANT 310: Architecting for Real-Time Insights with Amazon Kinesis | Thu 3:15 Sessions that already occurred: ANT 208, ANT322-R Introducing ML-powered insights with Amazon QuickSight | Wed 1:00pm | Aria East, Level 1, Joshua 9 ANT311: NFL and Forwood Safety Deploy Business Analytics at Scale with Amazon QuickSight | Fri 11:30 Amazon EMR Sessions that already occured: ANT309, ANT308Sessions from prior days: ANT324
  40. 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Customers tell us: they have three type of projects Quickly build new apps in the cloud Gain new insights “Lift and shift” existing apps to the cloud
  41. 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Characteristics of modern applications Internet-scale and transactional Users: 1M+ Data volume: TB–PB–EB Locality: Global Performance: Milliseconds–microseconds Request Rate: Millions Access: Mobile, IoT, devices Scale: Up-out-in Economics: Pay-as-you-go Developer access: Instant API accessSocial mediaRide hailing Media streaming Dating
  42. 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS databases services Purpose-built for all your app needs DynamoDB NeptuneRDS Aurora CommercialCommunity Timestream QLDBElastiCache Relational Key-value Document In-memory Graph Time series Ledger
  43. 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon DynamoDB Fast and flexible key value database service for any scale Comprehensive security Encrypts all data by default and fully integrates with AWS Identity and Access Management for robust security Performance at scale Consistent, single-digit millisecond response times at any scale; build applications with virtually unlimited throughput Global database for global users and apps Build global applications with fast access to local data by easily replicating tables across multiple AWS Regions Serverless No hardware provisioning, software patching, or upgrades; scales up or down automatically; continuously backs up your data
  44. 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DynamoDB powers the world’s largest applications
  45. 45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The need for transactions support Scenario: Customer purchases an item “PUT”: “TableName”: “Orders”, “OrderStatus”: “Sold”, “Item”: “Bike”, “Quantity”: “1” “PUT”: “TableName”: “Inventory”, “Item”: “Bike”, “Quantity”: “- 1” Transact-write-items { “PUT”: “TableName”: “Orders”, “OrderStatus”: “Sold”, “Item”: “Bike”, “Quantity”: “1” “PUT”: “TableName”: “Inventory”, “Item”: “Bike”, “Quantity”: “- 1” } If code fails here, we inserted the order but did not update inventory If code fails here, we updated some of our inventory, but not all Developer needs to write code to undo a partial operation Transaction support: The system ensures everything happens (or not) VS.
  46. 46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon DynamoDB Transactions (GA) Build internet-scale apps with ACID transactions Simplify application code with ACID guarantees Run transactions for large scale workloads Accelerate legacy migrations
  47. 47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DynamoDB—read/write capacity on-demand (GA) No more capacity planning—pay only for what you use No capacity planning No need to specify how much read/write throughput you expect to use Ideal for unpredictable workloads Ramp from zero to tens of thousands of requests per second on demand Pay only for what you use Pay-per-request pricing
  48. 48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Timestream (sign up for the preview) Fast, scalable, fully managed time series database 1,000x faster and 1/10th the cost of relational databases Collect data at the rate of millions of inserts per second (10M/second) Trillions of daily events Adaptive query processing engine maintains steady, predictable performance Analytics optimized for time series data Built-in functions for interpolation, smoothing, and approximation Serverless Automated setup, configuration, server provisioning, software patching
  49. 49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Quantum Ledger Database (QLDB) (Preview) Fully managed ledger database Track and verify history of all changes made to your application’s data Immutable Maintains a sequenced record of all changes to your data, which cannot be deleted or modified; you have the ability to query and analyze the full history Cryptographically verifiable Uses cryptography to generate a secure output file of your data’s history Easy to use Easy to use, letting you use familiar database capabilities like SQL APIs for querying the data Highly scalable Executes 2–3x as many transactions than ledgers in common blockchain frameworks
  50. 50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Managed Blockchain (preview) Create and manage scalable blockchain networks Choice of Hyperledger Fabric or Ethereum Hyperledger Fabric available today; Ethereum coming soon Fully managed Create blockchain networks with a few clicks; Manage them with simple API calls Easily analyze blockchain activity Easy to move data into QLDB for further analysis Scalable and secure Support thousands of client applications running millions of transaction; integrates with AWS KMS
  51. 51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon QLBD vs. Amazon Managed Blockchain Amazon QLDB Amazon Managed Blockchain Central trusted authority Track and verify transactions with centralized ownership Ledger is immutable and cryptographically verifiable Ledger is owned by a trusted and centralized authority No central trusted authority Execute transactions and contracts with decentralized ownership Ledger is immutable and cryptographically verifiable Each party maintains their own copy of the ledger Use Ethereum or Hyperledger Fabric to build blockchain networks
  52. 52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. More places to learn about purpose-built databases Building modern applications using DynamoDB Transactions | Wed 7:00pm | MGM, Level 3, Premier Ballroom 310 Sessions that already occurred: DAT321, DAT401, DAT201, DAT303, DAT314, DAT325, DAT320, DAT332 Amazon DynamoDB Amazon Neptune Amazon ElastiCache Amazon QLDB Amazon Managed Blockchain Sessions that already occurred: DAT202, DAT319, DAT302-R1, DAT302-R Sessions that already occurred: DAT403, DAT315, DAT316 Use Cases for Amazon QLDB | Fri 9:15 | Mirage, Montego D Deep Dive on Amazon Blockchain | Wed 3:15 | Venetian, Level 2, Titian 2205 - T2
  53. 53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Customers tell us: they have three type of projects Quickly build new apps in the cloud Gain new insights “Lift and shift” existing apps to the cloud
  54. 54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Database Migration Service M I G R A T I N G D A T A B A S E S T O A W S Migrate between on-premises and AWS Migrate between databases Automated schema conversion Data replication for zero downtime migration
  55. 55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Customers want to lift and shift to the cloud Relational databases Non-relational databases Data warehouses Hadoop and Spark Redshift EMR Operational analytics Elasticsearch ServiceAurora DynamoDB Business Intelligence QuickSightRDS
  56. 56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. >100,000 databases migrated with DMS
  57. 57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Customers are migrating their workloads to AWS Verizon is migrating over 1,000 business-critical applications and database backend systems to AWS, several of which also include the migration of production databases to Amazon Aurora. Wappa migrated from their Oracle database to Amazon Aurora and improved their reporting time per user by 75 percent. Trimble migrated their Oracle databases to Amazon RDS and project they will pay about 1/4th of what they paid when managing their private infrastructure. Intuit migrated from Microsoft SQL Server to Amazon Redshift to reduce data-processing timelines and get insights to decision makers faster and more frequently. Equinox Fitness migrated its Teradata on-premises data warehouse to Amazon Redshift. They went from static reports to a modern data lake that delivers dynamic reports. Eventbrite moved from Cloudera to Amazon EMR and were able to cut costs dramatically, spinning clusters up/down on-demand and using Spot (saving > 80%) and Reserved Instances. By December 2018, Amazon.com will have migrated 88% of their Oracle DBs (and 97% of critical system DBs) moved to Amazon Aurora and Amazon DynamoDB. They also migrated their 50 PB Oracle Data Warehouse to AWS (Amazon S3, Amazon Redshift, and Amazon EMR). Samsung Electronics migrated their Cassandra clusters to Amazon DynamoDB for their Samsung Cloud workload with 70% cost savings.
  58. 58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora MySQL and PostgreSQL compatible relational database built for the cloud Performance and availability of commercial-grade databases at 1/10th the cost Availability and durability Fault-tolerant, self-healing storage; six copies of data across three AZs; continuous backup to S3 Fully managed Managed by RDS: no hardware provisioning, software patching, setup, configuration, or backups Highly secure Network isolation, encryption at rest/transit Performance and scalability 5x throughput of standard MySQL and 3x of standard PostgreSQL; scale-out up to 15 read replicas
  59. 59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora Global Database (GA) High-performance database for globally-distributed applications Single Global Database with cross region replication Replication typically completes in less than a second No impact on database performance Write master in one region and read replicas in other regions Cross-region disaster recovery Local read latency for applications with global users Primary Region Secondary Region Application Storage Storage Replication <1s
  60. 60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon RDS Managed relational database service with a choice of six popular database engines Available & durable Automatic Multi-AZ data replication; automated backup, snapshots, failover Easy to administer No need for infrastructure provisioning, installing and maintaining DB software Highly scalable Scale database compute and storage with a few clicks with no application downtime Fast & secure SSD storage and guaranteed provisioned I/O; data encryption at rest and in transit
  61. 61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Databases in private data centers are still difficult and expensive to set up and manage Difficult to set up and manage databases for high availability across multiple nodes Personnel needed to create the database image, install operating system, packages, and setup Burdensome to support multiple versions and applying patching ?
  62. 62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon RDS on VMware (Preview) Managed service for on-premises databases RDS deployed as a service in on-premises VMware private data centers (vSphere) Automates management of on-premises databases and hybrid backup and scaling Available and durable Enable hybrid features and tap into AWS for high availability, backup, and restore Secure and compliant Automate management of databases for workloads that must remain on-premises to adhere to strict data policies Fully managed Easy to provision, monitor, and operate relational databases in your private data center Scalability and performance Scale storage, compute, and memory of on-premises databases from a single, simple interface
  63. 63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How it works Amazon RDS on VMware RDS on VMware Deploy RDS on VMware in your private data center and manage on- premises databases using a single RDS interface RDS interface Use the Amazon RDS console, APIs, or CLI to provision databases Log in or create AWS account Find RDS on VMware in the console and choose AWS region Download and install the connector Download & install the RDS connector in your VMware vSphere environment to establish secure VPN connection between AWS and your private data center Automate database management RDS on VMware automates database management tasks including provisioning, patching, backups and failover
  64. 64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. More places to learn about Aurora, RDS, and DMS Sessions that already occurred: DAT204-R1, DAT313, DAT318, DAT204-R, DAT304-R, DAT304-R1, DAT305-R, DAT305-R1, DAT336 Amazon Aurora DMS Amazon RDS Deep Dive on Amazon RDS on VMware | Thu 1:45 | Aria East, Plaza Level, Orovada 2 Chalk Talk on Amazon RDS on VMware | Fri 10:45 | Mirage, Martinique A DAT323: Best Practices for Running SQL Server on Amazon RDS | Thu 1:00 DAT402: Using Performance Insights to Optimize Database Performance | Thu 12:15 Sessions that already occurred: DAT203, DAT322, DAT324, DAT317 Sessions that already occurred: DAT207
  65. 65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. When to Use Which Services Situation Solution Existing application Use your existing engine on RDS • MySQL Amazon Aurora, RDS for MySQL • PostgreSQL Amazon Aurora, RDS for PostgreSQL • MariaDB Amazon Aurora, RDS for MariaDB • Oracle Use SCT to determine complexity Amazon Aurora, RDS for Oracle • SQL Server Use SCT to determine complexity Amazon Aurora, RDS for SQL Server New application • If you can avoid relational features DynamoDB • If you need relational features Amazon Aurora In-memory store/cache • Amazon ElastiCache Time series data • Amazon Timestream Track every application change, crypto verifiable. Have a central trust authority • Amazon Quantum Ledger Database (QLDB) Don’t have a trusted central authority • Amazon Managed Blockchain Data Warehouse & BI • Amazon Redshift, Amazon Redshift Spectrum, and Amazon QuickSight Adhoc analysis of data in S3 • Amazon Athena and Amazon QuickSight Apache Spark, Hadoop, HBase (needle in a haystack type queries) • Amazon EMR Log analytics, operational monitoring, & search • Amazon Elasticsearch Service and Amazon Kinesis
  66. 66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  67. 67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Architect services ground-up for the cloud and for the explosion of data Offer a portfolio of purpose-built services, optimized for your workloads Help you innovate faster through managed services Our approach Provide services that help you migrate existing apps and databases to the cloud
  68. 68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. EA is supporting their most mission critical workloads with Amazon Redshift. They were able to increase query throughput by 2x over the last 12 months.
  69. 69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Kinesis—real time Easily collect, process, and analyze video and data streams in real time Capture, process, and store video streams for analytics Load data streams into AWS data stores Analyze data streams with SQL Build custom applications that analyze data streams Kinesis Video Streams Kinesis Data Streams Kinesis Data Firehose Kinesis Data Analytics
  70. 70. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Build real-time streaming analytics in your Java apps Use Java Integrated Development Environment of choice Libraries include pre-built stream processing operators Real-time application can be built in hours Kinesis Data Analytics—Java support (GA) Devices Sensors IoT Kinesis Data Streams Kinesis Data Analytics Java IDE
  71. 71. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tailored recommendations to increase performance & reduce cost Redshift’s machine learning engine uncovers optimizations Operations such as vacuum and analyze run behind the scenes Available today Redshift ML based auto-tuning (GA) Clusters always optimized for best performance and lowest cost AUTO AUTO AUTO ADVISE ADVISE
  72. 72. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Most enterprise database & analytics cloud customers
  73. 73. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Most startup database & analytics cloud customers

×