Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deep Dive on Amazon Relational Database Service

777 views

Published on

Deep Dive on Amazon Relational Database Service

Published in: Technology
  • Be the first to comment

Deep Dive on Amazon Relational Database Service

  1. 1. ©  2016,  Amazon  Web  Services,  Inc.  or  its  Affiliates.  All  rights  reserved. 7  July  2016 Deep  Dive  on  Amazon  Relational   Database  Service Martin  Minnock,  Centre  for  Innovation  &  Analytics,  Aon Paul  Burne  -­ Technical  Account  Manager,  AWS Toby  Knight  -­ Manager,  Solutions  Architecture,  AWS
  2. 2. What  to  expect • Amazon  RDS  overview  (super  quick) • Security • Customer  story • Migrating  to  RDS • Metrics  and  monitoring • Scaling  on  RDS • Backups  and  snapshots • High  availability
  3. 3. No  infrastructure   management Scale  up/down Cost-­effective Instant  provisioning Application   compatibility Amazon  Relational   Database  Service  (Amazon  RDS)
  4. 4. Amazon  RDS  engines Commercial Open  source Amazon  Aurora
  5. 5. Amazon  Aurora  vs.  MySQL Feature RDS  Aurora RDS MySQL Number  of  replicas Up  to  15 Up  to  5 Replication  type Asynchronous   (milliseconds)   Asynchronous   (seconds) Replication  performance  impact  on   primary Low High Replica  can  act  as  failover  target Yes  (no  data  loss) Yes  (potentially  minutes  of  loss) Storage Up  to  64  TB,  auto  growth Up  to  6  TB,  specify  storage  limit Automated  failover Yes,  to  replica   Yes,  to  standby   User-­‐defined  replication  delay No Yes Replica  support  for  different  data  or   schema  vs.  primary No Yes Cross-­‐region  replication No Yes Data  cache  survives   Yes No
  6. 6. Trade-­offs  with  a  managed  service Fully  managed  host  and  OS • No  access  to  the  database  host  operating  system • Limited  ability  to  modify  configuration  that  is  managed  on  the   host  operating  system • No  functions  that  rely  on  configuration  from  the  host  OS Fully  managed  storage • Max  storage  limits • SQL  Server—4  TB • MySQL,  MariaDB,  PostgreSQL,  Oracle—6  TB • Aurora—64  TB • Growing  your  database  is  a  process
  7. 7. Selected  Amazon  RDS  customers
  8. 8. Security
  9. 9. Amazon  Virtual  Private  Cloud  (Amazon  VPC) Securely  control  network  configuration Availability  Zone AWS   Region 10.1.0.0/16 10.1.1.0/24 Manage  connectivity AWS  Direct   Connect VPN   Connection VPC   Peering Internet   Gateway Routing   Rules
  10. 10. Security  groups Database  IP  firewall  protection Protocol Port  Range Source TCP 3306 172.31.0.0/16 TCP 3306 “Application security  group” Corporate  address  admins Application  tier
  11. 11. Compliance Singapore  MTCS 27001/9001 27017/27018
  12. 12. MySQL  and  Oracle • SOC  1,  2,  and  3 • ISO  27001/9001 • ISO  27017/27018 • PCI  DSS • FedRamp • HIPAA  BAA • UK  government  programs • Singapore  MTCS Compliance SQL  Server  and  PostgreSQL • SOC  1,  2,  and  3 • ISO  27001/9001 • ISO  27017/27018 • PCI  DSS • UK  government  programs • Singapore  MTCS
  13. 13. SSL Available  for  all  six  engines Using  SSL  to  encrypt  a  connection  to  a  DB  instance mysql -h myinstance.c9akciq32.rds-eu-west-1.amazonaws --ssl-ca=rds-combined-ca-bundle.pem --ssl-verify-server-cert.com
  14. 14. At-­rest  encryption • DB  instance  storage • Automated  backups • Read  Replicas • Snapshots • Available  for  all  six  engines • No  additional  cost • Support  compliance  requirements
  15. 15. AWS  KMS  — RDS  standard  encryption Two-­tiered  key  hierarchy  using  envelope  encryption • Unique  data  key  encrypts  customer  data • AWS  KMS  master  keys  encrypt  data  keys Benefits: • Limits  risk  of  compromised  data  key • Better  performance  for  encrypting  large  data • Easier  to  manage  small  number  of  master  keys   than  millions  of  data  keys • Centralized  access  and  audit  of  key  activity Data  Key  1 Amazon   S3  Object Amazon   EBS   Volume Amazon   Redshift   Cluster Data  Key  2 Data  Key  3 Data  Key  4 Custom Application Customer  Master Key(s)
  16. 16. Enabling  encryption AWS  Command  Line  Interface  (AWS  CLI) aws  rds  create-­db-­instance  -­-­region  us-­west-­2  -­-­db-­instance-­identifier  sg-­cli-­test   -­-­allocated-­storage  20  -­-­storage-­encrypted -­-­db-­instance-­class  db.m4.large  -­-­engine  mysql   -­-­master-­username  myawsuser  -­-­master-­user-­password  myawsuser aws  rds  create-­db-­instance  -­-­region  us-­west-­2  -­-­db-­instance-­identifier  sg-­cli-­test1   -­-­allocated-­storage  20  -­-­storage-­encrypted    -­-­kms-­key-­id  xxxxxxxxxxxxxxxxxx   -­-­db-­instance-­class  db.m4.large  -­-­engine  mysql   -­-­master-­username  myawsuser   -­-­master-­user-­password  myawsuser
  17. 17. Amazon  RDS  +  AWS  KMS  useful  hints   • You  can  only  encrypt  on  new  database  creation • Encryption  cannot  be  removed • Master  and  Read  Replica  must  be  encrypted • Unencrypted  snapshots  cannot  be  restored  to  encrypted  DB • Cannot  restore  MySQL  to  Aurora  or  Aurora  to  MySQL • Cannot  copy  snapshots  or  replicate  DB  across  regions
  18. 18. IAM  governed  access You  can  use  AWS  Identity  and  Access  Management  (IAM)   to  control  who  can  perform  actions  on  RDS Users  and  DBAApplications DBA  and  Ops Your  database RDS Controlled  with  IAMControlled  with  database  grants
  19. 19. IAM  governed  access Policies "Action":  [ "rds:Describe*", "rds:ListTagsForResource", "ec2:DescribeAccountAttributes", "ec2:DescribeAvailabilityZones", "ec2:DescribeSecurityGroups", "ec2:DescribeVpcs”, "cloudwatch:GetMetricStatistics", "logs:DescribeLogStreams",   "logs:GetLogEvents" ], "Effect":  "Allow", "Resource":  "*" "Action":  [ "rds:*", "cloudwatch:DescribeAlarms", "cloudwatch:GetMetricStatistics", "ec2:DescribeAccountAttributes", "ec2:DescribeAvailabilityZones", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcs", "sns:ListSubscriptions", "sns:ListTopics",   "logs:DescribeLogStreams",   "logs:GetLogEvents"   ], "Effect":  "Allow", "Resource":  "*" Read  Only Full  Access
  20. 20. Prepared  by  Aon   Inpoint  |  July  2016   RDS  Deep  Dive Martin  Minnock  -­ Aon  Centre  for  Innovation  &  Analytics
  21. 21. 130+  staff Data  Analysts  |  Data  Scientists   |  Business  Analysts  |  IT  Development,   Database  &  Infrastructure  Specialists Platforms,  Projects  &  Services multi-­channel  web  portals  |  ad-­hoc   reporting  |  statistical  analysis  |   machine  learning  initiatives Dublin  Centre  for  Innovation  and  Analytics  at  the  heart  of  Aon  Inpoint Agile  Scrum 16  cross-­functional  teams   Agile  Scrum  &  Kanban 2  weekly  sprints  |  Incremental  releases Aon  Inpoint  &  ACIA  (Dublin)
  22. 22. ACIA  Reference  Architecture  for  Analytics Data  Transformation  &  AnalysisData  Lake  Ingestion Database File/Object   Storage Message   Channel consume Data  Warehouses Advanced  Analysis Mart Marts Mart Mart Analytics   Distribution Bespoke   Analysis Reports APIs Web  Portal Dashboards Application  Middleware OrchestrationData  Sources Transactional Systems Documents Public Sources Reference Data Logs SQL APIs JSON/ XML SFTP/ PUT Metadata Workflow  &  BatchMessaging Technology  Management MonitoringSecurity Backup  &  Recovery ITIL  Service   Management integrate Logging  &  Audit.
  23. 23. Drivers  for  AWS  Cloud  Adoption Performance  and  Productivity Poor  server  performance Re-­purposing/refreshing   hardware Capacity  planning  fails Cumbersome  work  practices Engagement   Focus  on  business  differentiation Promote  experimentation  &  fail-­ fast Drive  innovation Develop  careers Costs  and  Risks   Poor  utilisation Responsiveness   to  change Emerging  security  standards Ageing  hardware  /  EoL Separation  of  duties Platform  for  Growth Global  user  base   Data  increase  across  4V’s   Auto-­scaling  analytics   Democratisation  of  data Relentless  business  appetite
  24. 24. Backend   Databases  for: Analytics   Delivery Analytics   Engine New Products Lift  &  Shift   Targets Short-­Life  POC   systems Precedent  for   native  AWS   services How  ACIA  uses  RDS
  25. 25. Risk/View  – Analytics  Platform  for  Market  &  Risk  Insights Rapid  Updates,   Agile  delivery Customisable   Future-­ Proofed,   Flexible Focused  on   Self-­Service  &   Automation Highly   Available Resource   Intensive
  26. 26. Challenges  (and  Solutions) 3rd Party  ToolsDatabase  Refreshes Missing  Functionality EC2  (&  BA) RDS  in  the  Ecosystem AWS  DMS
  27. 27. Complete  Lift  &  Shift  – 100%  AWS Data  Lake  – feat.  S3,  EMR,  and  ECS   New  Product  Development RDS  for  PostgreSQL,  AWS  Lambda  for  Python Innovation!  Data  Science  &  Machine  Learning Intentions  for  the  Future  – RDS  and  Beyond
  28. 28. ©  Aon  plc  or  its  affiliates  ("Aon"). All  rights  reserved.   NOTE: Aon  does  not  provide  or  express  an  opinion  or  recommendation  regarding  any  matter  mentioned  in  this   presentation.The  recipient  understands  that  neither  Aon  nor  its  employees  makes  or  shall  make  any  representation  or   warranty  as  to  the  accuracy  or  completeness  of  any  information  contained  in  this  presentation. Aon  shall  not  have  any   liability  to  the  recipient  or  any  other  party  resulting  from  the  use  of  such  information  by  the  recipient  or  any  other  party. The  information  contained  in  this  presentation  may  not  be  reproduced  in  any  way  or  disseminated  to  any  other  party   without  the  prior  written  consent  of  Aon. Aon  has  endeavoured  to  ensure  that  this  presentation  is  free  of  any  virus  or  any  other  thing  that  would  affect  the   recipient’s  computer  system. However,  Aon  cannot  guarantee  the  security  status  of  this  presentation  when  accessed  by   the  reader  and  shall  not  have  any  liability  to  the  reader,  recipient  or  any  other  party  resulting  from  access  to  or  use  of  the   information  contained  herein. Disclaimer
  29. 29. Migrating  onto  RDS
  30. 30. Historically,  Migration  =  Cost,  Time Commercial  data  migration  and  replication  software Complex  to  setup  and  manage Legacy  schema  objects,  PL/SQL  or  T-­SQL  code Application  downtime
  31. 31. Database  Migration  – 2  Steps
  32. 32. Step  1:  Schema  Conversion  Overview
  33. 33. ü Move  data  to  the  same  or  different  database  engine   ü Keep  your  apps  running  during  the  migration ü Start  your  first  migration  in  10  minutes  or  less ü Replicate  within,  to,  or  from  Amazon  EC2  or  RDS AWS  Database   Migration  Service
  34. 34. Customer premises Application  Users AWS Internet VPN Start  a  replication  instance Connect  to  source  and  target   database Select  tables,  schemas,  or   databases Let  the  AWS  Database  Migration   Service  create  tables,  load  data,   and  keep  them  in  sync Switch  applications  over  to  the   target  at  your  convenience Keep  your  apps  running  during  the  migration
  35. 35. Flexible  Migration  Approach Replication instance Source Target Target Target Multiple  targets Replication instance Source Target Source Source Multiple  sources   Source L Target Replication instance instance Selective
  36. 36. Metrics  and  monitoring
  37. 37. Summary  of  Metrics  and  Monitoring   • Amazon  RDS  Metrics • Event  Notifications • Log  Files • Cloudtrail
  38. 38. Accessing  Amazon  RDS  Metrics
  39. 39. Amazon  RDS  Standard  Metrics 45  MetricsChange  Time  Period Dive  Deeper Create   Alarms
  40. 40. Amazon  RDS  Enhanced  Monitoring Access  to  over  50  metrics  in  7   categories: • Memory,   • I/O,   • CPU,   • File  system,   • Load,   • Swap • Processes  
  41. 41. Amazon  RDS  Event  Notifications • Get  Notified  when  events  occur  on   your  database  instances • 17  different  event  categories   (availability,  backup,  configuration   change,  and  so  on) • Uses  Amazon  Simple  Notification   Service  (Amazon  SNS)  
  42. 42. Scaling  on  RDS
  43. 43. Scale  out  with  Read  Replicas Relieve  pressure  on  your  master   node  for  supporting  reads  and   writes. Bring  data  close  to  your  customer’s   applications  in  different  regions Promote  a  Read  Replica  to  a   master  for  faster  recovery  in  the   event  of  disaster Replicas  within  and  cross-­ region • MySQL,  MariaDB,   PostgreSQL • Aurora Engines  Needing   Other  Tools • Oracle   • Microsoft  SQL  Server
  44. 44. Creating  and  Prompting  Read  Replicas   Read  Replica  creation   and  promotion  are   accessed  from  the   Instance  Actions  button   in  the  RDS  console
  45. 45. Creating  and  Promoting  Read  Replicas  
  46. 46. Creating  and  Promoting  Read  Replicas  With  CLI  
  47. 47. Creating  and  Promoting  Read  Replicas  With  CLI  
  48. 48. Scaling  Up  and  Down • Handle  higher  load  or  lower  usage • Control  costs
  49. 49. Scaling  Up  and  Down Console
  50. 50. Backups  and  snapshots
  51. 51. RDS  Backups MySQL,  PostgreSQL,  MariaDB,  Oracle,  SQL  Server • Scheduled  daily  backup  of  entire  instance • Archive  database  change  logs • Up  to  35  day  retention  for  backups • I/O  suspension  as  backup  is  initiated  (but  not  with  multi-­AZ  deployment) • Multiple  copies  in  each  AZ  where  you  have  instances  for  a  deployment Aurora • Automatic,  continuous,  incremental  backups • Point-­in-­time  restore • No  impact  on  database  performance • 35  day  retention
  52. 52. RDS  Restore • Restoring  creates  an  entire  new  database  instance • You  define  all  the  instance  configuration  just  like  a  new   instance
  53. 53. Snapshots • Full  copies  of  your  Amazon  RDS  database  that  are  different  from   your  scheduled  backups • Backed  by  Amazon  S3 • Typical  use  cases • Resolve  production  issues • Nonproduction  environments • Point-­in-­time  restore • Final  copy  before  terminating  a  database • Disaster  recovery • Cross-­region  copy • Copy  between  accounts
  54. 54. High  availability
  55. 55. Minimal  deployment—single  AZ Availability  Zone AWS   Region 10.1.0.0/16 10.1.1.0/24 Amazon  Elastic  Block  Store   Volume
  56. 56. High  availability—Multi-­AZ Availability  Zone  A AWS   Region 10.1.0.0/16 10.1.1.0/24 Availability  Zone  B 10.1.2.0/24 Replicated  storage Same  instance   type  as  master
  57. 57. High  availability—Multi-­AZ  to  DNS dbinstancename.1234567890.us-­west-­2.rds.amazonaws.com:3006
  58. 58. High  availability—Amazon  Aurora  storage • Storage  volume  automatically  grows  up  to   64 TB • Quorum  system  for  read/write;;  latency   tolerant • Peer-­to-­peer  gossip  replication  to  fill  in   holes • Continuous  backup  to  Amazon  S3  (built  for   11 9s  durability) • Continuous  monitoring  of  nodes  and  disks   for  repair   • 10  GB  segments  as  unit  of  repair  or  hotspot   rebalance • Quorum  membership  changes  do  not  stall   writes AZ  1 AZ  2 AZ  3 Amazon S3
  59. 59. High  availability—Aurora  nodes • Aurora  cluster  contains  primary   node  and  up  to  15  secondary   nodes • Failing  database  nodes  are   automatically  detected  and   replaced • Failing  database  processes  are   automatically  detected  and  recycled • Secondary  nodes  automatically   promoted  on  persistent  outage,  no   single  point  of  failure • Customer  application  can  scale  out   read  traffic  across  secondary  nodes AZ  1 AZ  3AZ  2 Primary Node Primary Node Primary Node Primary Node Primary Node Secondary Node Primary Node Primary Node Secondary Node
  60. 60. Aurora-­DNS  Failover App RunningFailure  Detection DNS  Propagation Recovery Recovery DB Failure MYSQL App Running Failure  Detection DNS  Propagation Recovery DB Failure AURORA  WITH  MARIADB  DRIVER 1 5 -­ 3 0   s e c 5 -­ 2 0   s e c 1 5 -­ 3 0   s e c Driver  benefits
  61. 61. Thank  You!
  62. 62. Contacts Martin  Minnock Cloud  Product  Owner  &  Database  Manager   Aon  Centre  for  Innovation  &  Analytics martin.minnock@aon.ie Paul  Burne Technical  Account  Manager Amazon  Web  Services paulburn@amazon.co.uk Toby  Knight Manager,  Solutions  Architecture Amazon  Web  Services tobyk@amazon.co.uk @martinminnock
  63. 63. Please  remember  to  rate  this   session  under  My  Agenda  on   awssummit.london

×