Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Build data-drive, high performance, internet scale applications with AWS Databases

125 views

Published on

In this session, we share database architecture strategies and best practices for building high-performance and internet-scale applications using Amazon DynamoDB, Amazon Timestream, and Amazon ElastiCache. We discuss the various types of modern applications and tell you which AWS services are the best for each use case. For globally distributed applications that need to scale to petabytes of storage, we discuss the AWS database services and design patterns that are tailored for internet-scale applications. We also discuss how to analyze time-series data from modern applications. Finally, we cover speed-intensive application types that require real-time microsecond responsiveness, and we explain how to design your data models to meet these rigorous requirements.

  • Be the first to comment

  • Be the first to like this

Build data-drive, high performance, internet scale applications with AWS Databases

  1. 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Build data driven, high-performance, internet- scale applications with AWS databases Bastien Leblanc Solutions Architect Amazon Web Services @bastleblanc bastil@amazon.co.uk D A T 2 Philip McMahon Software Developer The Guardian @pmcmahon philip.mcmahon@theguardian.com
  2. 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T What are modern Internet scale applications Purpose built Databases for Internet scale Why and How The Guardian moved to RDS Real time applications Agenda
  3. 3. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  4. 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Characteristics of modern applications Internet-scale and transactional Users: 1M+ Data volume: TB–PB–EB Locality: Global Performance: Milliseconds–microseconds Request Rate: Millions Access: Mobile, IoT, devices Scale: Up-out-in Economics: Pay-as-you-go Developer access: Instant API accessSocial mediaRide hailing Media streaming Dating
  5. 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Modern applications require internet-scale performance
  6. 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Internet Scale Reference Architecture AWS Cloud Global Users Amazon S3 static assets https Amazon CloudFront Amazon Route 53 AWS WAF AWS Lambda Amazon API Gateway Main Region Data Layer Instances Instances Load Balancing (NLB/ALB) 2nd Region Amazon S3 static assets Database(s) ReplicasAWS LambdaLoad Balancing (NLB/ALB) AWS Cloud
  7. 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Caching and distribution at the edge / AWS Cloud Global Users Amazon S3 static assets https Amazon CloudFront Amazon API Gateway Main Region Load Balancing (NLB/ALB) AWS WAF AWS Lambda@Edge
  8. 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Internet Scale Reference Architecture Scaling best practices at the compute layer - Autoscaling : Find the right metric (hint: think business) - Serverless - Asynchronous call as much as possible - Concurrency: think AWS Lambda execution time - Decouple : - SQS - Kinesis - Step Functions AWS Cloud AWS Lambda Main Region EC2 Instances Load Balancing ALB or NLB Amazon ECS ContainersAmazon EKS AWS Fargate AutoscalingAutoscaling
  9. 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Internet Scale Reference Architecture What about the Data layer?
  10. 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Common data categories and use cases Relational Referential integrity, ACID transactions, schema- on-write Lift and shift, ERP, CRM, finance Key-value High throughput, low- latency reads and writes, endless scale Real-time bidding, shopping cart, social, product catalog, customer preferences Document Store documents and quickly access querying on any attribute Content management, personalization, mobile In-memory Query by key with microsecond latency Leaderboards, real-time analytics, caching Graph Quickly and easily create and navigate relationships between data Fraud detection, social networking, recommendation engine Time-series Collect, store, and process data sequenced by time IoT applications, event tracking Ledger Complete, immutable, and verifiable history of all changes to application data Systems of record, supply chain, health care, registrations, financial
  11. 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Purpose-built databases Relational Key-value Document In-memory Graph Time-series Ledger Amazon DynamoDB Amazon Neptune Amazon RDS Aurora CommercialCommunity Amazon Timestream Amazon Quantum Ledger Database Amazon ElastiCache Amazon DocumentDB with MongoDB
  12. 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Customers Amazon API Gateway Authorizer Amazon EC2 Manage Static Content Amazon CloudFront Stream Amazon Elasticsearch Service Customer Data Amazon DynamoDB Fender: Teaching How to Play a Guitar with Serverless Technologies Get Admin - Data driven - Internet scale - Going global Mobile app Fender Play allows any person to learn how to play a guitar or Ukulele by leveraging AWS serverless technologies Photo by Katie Montgomery on Unsplash This is My Architecture Video: https://youtu.be/RzRVwh_pXN4
  13. 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.SUMMIT The Guardian - Migrating from MongoDB to Postgres RDS Philip McMahon Software Developer The Guardian DAT2
  14. 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 20+ Microservices Scala/React/Angular EC2/Lambda Dynamo DB/RDS/Elasticsearch/S3 Editorial Tools at The Guardian
  15. 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
  16. 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 100% uptime required ‘Source of truth’ for all content Encryption at rest essential Not responsible for serving content Composer
  17. 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
  18. 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
  19. 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Why we left Mongo
  20. 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Preference for ‘PaaS’ DB in AWS DynamoDB - only 5 indexes, no encryption at rest Postgres RDS used in other tools JSONB support in Postgres Why RDS
  21. 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Schema…? CREATE TABLE content( id VARCHAR(50) PRIMARY KEY, content jsonb );
  22. 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T UPDATE draft_content SET content = (jsonb_insert( content, {tags,keyword,-1}, {' "id": "politics/brexit", "name": "Brexit" '}::jsonb, insert_after=true)) WHERE {'"id": "politics/brexit", "name": "Brexit"'}::jsonb NOT IN (SELECT * FROM jsonb_array_elements((content #>> {tags,keyword})::jsonb)) AND id = "abc123"; Working with JSONB
  23. 23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Syntax has to match exactly Type conversions possible Any depth possible JSONB indexes CREATE INDEX last_modified_index ON content(((content #>>'{contentChangeDetails,lastModified,date}')::BIGINT));
  24. 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Second API Proxy to keep new API up to date Separate migration script GOR for load testing Migration strategy
  25. 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
  26. 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
  27. 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Switching over
  28. 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Proxy became unreliable Dealing with 2 APIs added cost to making changes ‘Primary/Secondary’ setup complex to manage Challenges
  29. 29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Double API setup not ideal JSONB works but syntax is a challenge, indexes essential Postgres is great Learnings
  30. 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Mongo/EC2: - DB instances: 3 x r3.xlarge = $14,200 - OpsManager: 1.5 x r3.xlarge = $7,100 = $21000 (+ licensing) Postgres/RDS: - db.r4.xlarge with High Availability = $6000 Cost (annually)
  31. 31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T IAM authentication Aurora option Cross region snapshot replication (Almost) no more DB Admin Nice features
  32. 32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T But...
  33. 33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Thank you! SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. We’re hiring! workforus.theguardian.com
  34. 34. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  35. 35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Internet-scale Apps Need Low Latency and High Concurrency Users 1M+ Data volume TB-PB-EB Locality Global Performance Milliseconds -Microseconds Request Rate Millions Access Mobile, IoT, Devices Scale Up-Out-In Economics Pay as you go Developer access Instant API access Gaming leaderboards Financial trading Social media Ride hailing Media streaming Dating Session stores
  36. 36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Introducing Amazon ElastiCache Fully-managed, Redis or Memcached compatible, low-latency, in-memory data store Fully Managed Extreme Performance Easily Scalable AWS manages all hardware and software setup, configuration, monitoring In-memory data store and cache for sub-millisecond response times Read scaling with replicas. Write and memory scaling with sharding. Non disruptive scaling
  37. 37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Caching with Elasticache Clients Amazon ElastiCache Redis Amazon DynamoDB Elastic Load Balancing Amazon EC2 Amazon RDS write-through reads/ writes DDB streams mysql.lambda_async reads/ writes Amazon S3 reads/writes Object data Unstructured data Relational data
  38. 38. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  39. 39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T When to Use Which Services Situation Solution Existing application Use your existing engine on RDS • MySQL Amazon Aurora, RDS for MySQL • PostgreSQL Amazon Aurora, RDS for PostgreSQL • MariaDB Amazon Aurora, RDS for MariaDB • Oracle Use SCT to determine complexity Amazon Aurora, RDS for Oracle • SQL Server Use SCT to determine complexity Amazon Aurora, RDS for SQL Server New application • If you can avoid relational features DynamoDB • If you need relational features Amazon Aurora In-memory store/cache • Amazon ElastiCache Time series data • Amazon Timestream Track every application change, crypto verifiable. Have a central trust authority • Amazon Quantum Ledger Database (QLDB) Don’t have a trusted central authority • Amazon Managed Blockchain Data Warehouse & BI • Amazon Redshift, Amazon Redshift Spectrum, and Amazon QuickSight Adhoc analysis of data in S3 • Amazon Athena and Amazon QuickSight Apache Spark, Hadoop, HBase (needle in a haystack type queries) • Amazon EMR Log analytics, operational monitoring, & search • Amazon Elasticsearch Service and Amazon Kinesis
  40. 40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Caching : at the edge / locally Scale: serverless or containers Purpose built Databases Internet Scale Applications with AWS Go Global
  41. 41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Thank you! S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Bastien Leblanc Solutions Architect Amazon Web Services @bastleblanc bastil@amazon.co.uk Philip McMahon Software Developer The Guardian @pmcmahon philip.mcmahon@theguardian.com
  42. 42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Photo by Daniel Fontenele on Unsplash Photo by Glenn Carstens-Peters on Unsplash

×