Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Relational Databases Utilising Amazon RDS - Technical 201

511 views

Published on

How can organisations leverage the cloud for running relational database workloads? In this session we cover architectures and best practices that enable high performance, high availability, and graceful migration for relational database workloads while reducing costs. Hear how customers are using Amazon RDS to manage their databases at scale. This session teaches you to take advantage of features unique to AWS and Amazon RDS to free your databases from the confines of the conventional data centre.

Speaker: Adrian Hunter, Solutions Architect, Amazon Web Services

Featured Customer - Suncorp

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Relational Databases Utilising Amazon RDS - Technical 201

  1. 1. ©  2016,  Amazon  Web  Services,  Inc.  or  its  Affiliates.  All  rights  reserved. Adrian  Hunter,  Solution  Architect Amazon  Web  Services Sam  Johnston,  BI  Cloud  Architect Suncorp Relational  Databases  Utilising Amazon  RDS Technical  201
  2. 2. Overview  of  Amazon  RDS PerformanceReliabilitySecurity Cost   Optimisation Migration Suncorp’s Story Live  Demo
  3. 3. Overview  of  Amazon  RDS
  4. 4. Power,  HVAC,  net Rack  &  stack Server  maintenance OS  patches DB  s/w  patches Database  backups Scaling High  availability DB  s/w  installs OS  installation Query  Construction Query  Optimisation Schema  Design Why  Managed  Databases? Traditional  DC
  5. 5. Power,  HVAC,  net Rack  &  stack Server  maintenance OS  patches DB  s/w  patches Database  backups Scaling High  availability DB  s/w  installs OS  installation Query  Construction Query  Optimisation Schema  Design Why  Managed  Databases? Power,  HVAC,  net Rack  &  stack Server  maintenance OS  patches DB  s/w  patches Database  backups Scaling High  availability DB  s/w  installs OS  installation Query  Construction Query  Optimisation Schema  Design Traditional  DC DB  on  EC2
  6. 6. Power,  HVAC,  net Rack  &  stack Server  maintenance OS  patches DB  s/w  patches Database  backups Scaling High  availability DB  s/w  installs OS  installation Query  Construction Query  Optimisation Schema  Design Why  Managed  Databases? Power,  HVAC,  net Rack  &  stack Server  maintenance OS  patches DB  s/w  patches Database  backups Scaling High  availability DB  s/w  installs OS  installation Query  Construction Query  Optimisation Schema  Design Power,  HVAC,  net Rack  &  stack Server  maintenance OS  patches DB  s/w  patches Database  backups Scaling High  availability DB  s/w  installs OS  installation Query  Construction Query  Optimisation Schema  Design Traditional  DC DB  on  EC2 Amazon  RDS
  7. 7. Relational  Databases Fully  managed Simple  and  fast  to  scale Fast,  predictable  performance Low  cost,  pay  for  what  you  use Amazon RDS Amazon Aurora
  8. 8. Security
  9. 9. Network  Layer Amazon  VPC:  Control  subnets,  AZ   specificity  (DB  subnet  groups),  route   tables  and  NACLs Security  Groups:  Restrict  instance   traffic Public  Access:  Avoid  it  or  limit  it 1 2 3
  10. 10. Instance  Access Control:  Use  AWS  Identity  and  Access   Management  (IAM)  to  control  instance   lifecycle  permissions,  grant  least   privileges Audit:  Use  AWS  CloudTrail  to  log  AWS   API  invocations 1 2
  11. 11. Data  Access Grant  least  privileges  to  applications  and   end  users Amazon  RDS:  Database  level   authentication 1 2
  12. 12. Encryption Protect  data  at  rest:  Client  side   application,  TDE,  Encrypted  DB   instances  using  AWS  KMS Secure  data  in  transit:  Encrypted   connections  via  SSL 1 2
  13. 13. Reliability
  14. 14. Backup  and  Recovery Automated  Backups • Nightly  system  snapshots  +  transaction  backup • Enables  point-­in-­time  restore  to  any  point  in   retention  period,  up  to  the  last  5  minutes • Max  retention  period  =  35  days DB  Snapshots • User-­driven  snapshots  of  database • Kept  until  explicitly  deleted   1 2
  15. 15. Admin  Windows Configurable  administrative  windows  of  time Backup  window at  least  30  minutes,  once  a  day Maintenance  window at  least  30  minutes,  once  a  week 1 2
  16. 16. SecondaryPrimary AZ1 AZ2 Application Availability  – Multi-­AZ
  17. 17. SecondarySecondary Physical Synchronous Replication Primary AZ1 AZ2 Application Availability  – Multi-­AZ
  18. 18. SecondarySecondary Physical Synchronous Replication Primary AZ1 AZ2 Application Application Availability  – Multi-­AZ
  19. 19. SecondarySecondaryPrimary AZ1 AZ2 Application Application Availability  – Multi-­AZ
  20. 20. SecondarySecondary PrimaryPrimary AZ1 AZ2 Application Application Availability  – Multi-­AZ
  21. 21. SecondarySecondary PrimaryPrimary AZ1 AZ2 DNS Application Application Primary  Update Availability  – Multi-­AZ
  22. 22. SecondarySecondary PrimaryPrimary AZ1 AZ2 DNS cname update   Application Application Primary  Update Availability  – Multi-­AZ
  23. 23. SecondarySecondary PrimaryPrimary AZ1 AZ2 DNS cname update   Application Application Primary  Update Availability  – Multi-­AZ
  24. 24. SecondarySecondary Primary Physical Synchronous Replication AZ1 AZ2 DNS cname update   Application Application Primary  Update Availability  – Multi-­AZ
  25. 25. Minimise Unavailability  During  Scaling 4:30:58  PM  …  4:56:17  PM  :  25  minutes   4:43:31  PM  …  4:45:16  PM  :  105  seconds m3.large  (2  vCPU,  7.5GiB  RAM)  >>  m3.xlarge  (4  vCPU,  15GiB  RAM)  
  26. 26. Performance
  27. 27. Read  Replicas  for  Scale Secondary Application Secondary Sync Replication Multi-­AZ Primary
  28. 28. Read  Replicas  for  Scale Secondary Application Read   Replica Read   Replica Secondary Sync Replication Multi-­AZ Primary Async Replication
  29. 29. Read  Replicas  for  Scale Secondary Read   Replica Read   Replica Secondary Sync Replication Multi-­AZ Primary Eventually  Consistent   Reads Writes  &   Consistent   Reads Async Replication
  30. 30. Read  Replicas  Improve  Availability Secondary Read   Replica Read   Replica Secondary Sync Replication Multi-­AZ Primary Eventually  Consistent   Reads Writes  &   Consistent   Reads Async Replication
  31. 31. Read  Replicas  Improve  Availability Secondary Read   Replica Read   Replica Secondary Primary Sync Replication Multi-­AZ Eventually  Consistent   Reads Writes  &   Consistent   Reads Async Replication
  32. 32. SecondarySecondary Primary Upgrade Eventually  Consistent   Reads Writes  &   Consistent   Reads Async Replication Read  Replicas  Improve  Availability Read   Replica Read   Replica
  33. 33. Read  Replicas  Improve  Availability Secondary Read   Replica Read   Replica Secondary Primary Sync Replication Multi-­AZ Eventually  Consistent   Reads Writes  &   Consistent   Reads Async Replication Modify
  34. 34. Sizing  to  meet  your  Workload  Demands What  is  your  Workload’s  Demand  Pattern? Constant,  steady-­state Predictable  fluctuations,  steady-­state Growing,  but  predictable Fluctuating,  spiky What  is  your  Workload’s  Query  Pattern? For  example:  simple  key  lookups For  example:  complex  analytics    
  35. 35. Workload  Pattern  Impact  on  Performance Compute Capabilities vCPUs Memory Capabilities GB  of  RAM Network Performance MB/s  (Throughput) Storage Performance I/O  Throughput RDS  DB  Instance  Class RDS  Storage  Type
  36. 36. Compute  Resources  – Scaling  and  Performance Range  of  DB   instance  types From:  1  vCPU  and   1  GB  of  RAM To:  40  vCPUs  and   244  GB  of  RAM Grouped  in  instance   families: Standard Memory  Optimized Burstable  Capacity Scale  up  or  down  by   changing  the   instance  class R3
  37. 37. RDS  Storage  Types Magnetic General  Purpose (SSD) Provisioned  IOPS   (SSD)
  38. 38. Monitoring  and  Alerting Monitor  performance  using  Amazon  CloudWatch Default  Metrics and  Enhanced  Monitoring Alarms  &   notifications 1 2
  39. 39. Cost  Optimisation
  40. 40. Environments SyM A A RRRR A A Production
  41. 41. Environments SyM A A RRRR A A Production SyM A A RRRR A A Pre-­Production
  42. 42. Environments SyM A A RRRR A A SyM A A RRRR A A Production Pre-­Production
  43. 43. Environments SyM A A RRRR A A SyM A A RRRR A A Production Pre-­Production UAT yM A A RR
  44. 44. Environments SyM A A RRRR A A SyM A A RRRR A A yM A yM A yM A yM A yM A yM A Production Pre-­Production Dev  /  Test UAT yM A A RR
  45. 45. Environments SyM A A RRRR A A SyM A A RRRR A A yM A yM A yM A yM A yM A yM A Production Pre-­Production Dev  /  Test UAT yM A A RR
  46. 46. Burst  Mode  – T2  and  GP2 T2  – Instance  Family  with  Burst  Capability • Base  performance  +  burst   • Earn  credits  per  hour  when  below  base  performance • Can  store  up  to  24  hours’  worth  of  credits • Amazon  CloudWatch metrics  to  see  credits  and  usage GP2  – SSD  based  Amazon  EBS  Storage • 3  IOPS  per  GB  base  performance • Earn  credits  when  usage  below  base • Burst  to  3000+  IOPS  
  47. 47. Burst  Mode  – T2  and  GP2 6000  tps 3800  tps 2200  tps GP2  exhaust  credits T2  exhaust  CPU  credits CPU   Utilisation ReadIOPS
  48. 48. Reserved  Instances • Significant  discount  compared  to  On  Demand  rates • 1  Year  or  3  Year  options • Payment  options • All  Upfront • Partial  Upfront • No  Upfront • No  changes  in  the  way  you  operate
  49. 49. Lean Reporting as a Service Sam Johnston BI Cloud Architect
  50. 50. Suncorp Group:  A  top  20  ASX-­listed  company,  largest  insurer  (by  gross  written  premium),  5th largest  bank,   relationships  with  9  million+  customers,  $96  billion  in  Assets  under  management,  14,500+  employees
  51. 51. Requirements Regulatory  &  Contractual  Compliance Data  Quality Data-­Use  Approval   Data-­Use  Auditability Self  Service Regulatory  &  Contractual  Compliance Data  Quality Performance  Consistency Cost  Approval  &  Transparency Self  Service Platform  Owner Data  Security   Performance  Consistency   Cost  Transparency Lowest  Platform  Support  Overhead Ease  of  Maintenance Report  Developer Fast  Dev/Test  Environment  Delivery Easier  Access  To  Data  Needed Performance  Consistency   Improved  Productivity  (Performant   Env) Self  Service Data  Owner Report  Owner
  52. 52. Requirements Regulatory  &  Contractual  Compliance Data  Quality Data-­Use  Approval Data-­Use  Auditability Self  Service Regulatory  &  Contractual  Compliance Data  Quality Performance  Consistency Cost  Approval  &  Transparency Self  Service Platform  Owner Data  Security   Performance  Consistency   Cost  Transparency Lowest  Platform  Support  Overhead Ease  of  Maintenance Report  Developer Fast  Dev/Test  Environment  Delivery Easier  Access  To  Data  Needed Performance  Consistency   Improved  Productivity  (Performant   Env) Self  Service Data  Owner Report  Owner
  53. 53. Requirements Regulatory  &  Contractual  Compliance Data  Quality Data-­Use  Approval Data-­Use  Auditability Self  Service Regulatory  &  Contractual  Compliance Data  Quality Performance  Consistency Cost  Approval  &  Transparency Self  Service Platform  Owner Data  Security   Performance  Consistency   Cost  Transparency Lowest  Platform  Support  Overhead Ease  of  Maintenance Report  Developer Fast  Dev/Test  Environment  Delivery Easier  Access  To  Data  Needed Performance  Consistency Improved  Productivity  (Performant   Env) Self  Service Data  Owner Report  Owner
  54. 54. Requirements Regulatory  &  Contractual  Compliance Data  Quality Data-­Use  Approval Data-­Use  Auditability Self  Service Regulatory  &  Contractual  Compliance Data  Quality Performance  Consistency Cost  Approval  &  Transparency Self  Service Platform  Owner Data  Security   Performance  Consistency   Cost  Transparency Lowest  Platform  Support  Overhead Ease  of  Maintenance Report  Developer Fast  Dev/Test  Environment  Delivery Easier  Access  To  Data  Needed Performance  Consistency   Improved  Productivity  (Performant   Env) Self  Service Data  Owner Report  Owner
  55. 55. Reporting  – Views  of  Data Old  Way • Inter-­dependent,  multi-­stage  ETL • Cascading  Failures  /  Slowness • Contention • Complex  =  inertia Domino  Effect. Photograph  by  Martin  Fisch.  Unmodified. This  work  is  licensed  under   a Creative  Commons  Attribution   4.0   International   License.
  56. 56. Reporting  – Views  of  Data New  Way • One  platform  per  Data  View • Isolates  failure • Size  compute  to  job • Consistent  performance • Lean  – reduce  waste,  focus  on  value Old  Way • Inter-­dependent,  multi-­stage  ETL • Cascading  Failures  /  Slowness • Contention • Complex  =  inertia
  57. 57. Lean  Reporting  as  a  Service Data   Lake System  of   Record System  of   Record System  of   Record Data Flow Data   Owner(s) Multiple  Source  Systems Centralised Data  Store Data  Governance
  58. 58. Batch   Tier RDS SQL LAKEView Data   Lake System  of   Record System  of   Record System  of   Record Data Flow Data Flow Data   Owner(s) Report Owner(s) Batch  Workload Subset  of  Data SQL  Code Lean  Reporting  as  a  Service
  59. 59. Batch   TierServing  Tier Report Visualisation RDS View Rep Data   Lake System  of   Record System  of   Record System  of   Record Data Flow Data Flow Data Flow Data Flow Data   Owner(s) Report Owner(s) Report Users Data  View  Persists Batch  Tier  Disposed Approved  Access  Only Lean  Reporting  as  a  Service
  60. 60. Batch   Tier RDS SQL LAKEView Serving  Tier Report Visualisation RDS View Rep Data   Lake System  of   Record System  of   Record System  of   Record Data Flow Data Flow Data Flow Data Flow Time To  Live <  24Hrs TTL  <  7  Hrs Data   Owner(s) Report Owner(s) Report Users Variable  Service  Hours Stateless  Serving  Tier Lean  Reporting  as  a  Service
  61. 61. Batch   Tier RDS SQL LAKEView Serving  Tier RDS View Rep Data   Lake System  of   Record System  of   Record System  of   Record Data Flow Data Flow Data Flow Data Flow Report Users Batch   Tier RDS SQL LAKEView Serving  Tier RDS View Rep Data Flow Data Flow Data Flow Report Users Batch   Tier RDS SQL LAKEView Serving  Tier Report Visualisation RDS View Rep Data Flow Data Flow Data Flow Scale Out Data   Owner(s) Report Owner(s) Report Users Lean  Reporting  as  a  Service Report Visualisation Report Visualisation
  62. 62. Suncorp  Data  Service Batch   Tier RDS SQL LAKEView Serving  Tier Report Visualisation RDS View Rep Data   Lake System  of   Record System  of   Record System  of   Record Data Flow Data Flow Data Flow Data Flow Data   Owner(s) Report Owner(s) Report Users Batch   Tier RDS SQL LAKEView Serving  Tier Report Visualisation RDS View Rep Data Flow Data Flow Data Flow Report Users Batch   Tier RDS SQL LAKEView Serving  Tier Report Visualisation RDS View Rep Data Flow Data Flow Data Flow Scale Out Report Users Lean  Reporting  as  a  Service
  63. 63. Batch   Tier RDS SQL LAKEView Serving  Tier Report Visualisation RDS View Rep Data   Lake System  of   Record System  of   Record System  of   Record Data Flow Data Flow Data Flow Data Flow Self  Service  Portal Cost  Dashboard Governance  &  Control  – Ownership Data   Owner(s) Report Owner(s) • Notification  and  approval • Visibility • Reporting  – usage,  cost Report Users Lean  Reporting  as  a  Service
  64. 64. MonitorSuncorp  Data  Service Batch   Tier RDS SQL LAKEView Serving  Tier Report Visualisation RDS View Rep Data   Lake System  of   Record System  of   Record System  of   Record Data Flow Data Flow Data Flow Data Flow Move  DataMeta-­Data  Repo   Code  Repo DestroyBuild Data   Owner(s) Report Owner(s) Governance  &  Control  – System   Building Report Users Alarm • Infrastructure   +  Transform  as  Code • Build/Destroy • Move  approved  data  only • Monitoring/Enforcement Lean  Reporting  as  a  Service
  65. 65. Suncorp  Data  Service Data   Lake System  of   Record System  of   Record System  of   Record Data Flow Data Flow Data Flow Development Report Visualisation RDS LAKESQLViewRep Data   Owner(s) Report Owner(s) Report Developer Self  Service Controlled  Development  Access Scalability  per  Need Lean  Reporting  as  a  Service
  66. 66. Report Tester Suncorp  Data  Service Data   Lake System  of   Record System  of   Record System  of   Record Data Flow Data   Owner(s) Report Owner(s) Test  Batch   Tier RDS SQL LAKEView Test  Serving  Tier Report Visualisation RDS View Rep Data Flow Data Flow Data Flow Self  Service Controlled  Test  access Scalability  and  Operational  Readiness  Testing Lean  Reporting  as  a  Service
  67. 67. Lessons  Learned Leverage  Managed  Services Cost  Awareness Automate   Experiment.  Learn.  Iterate.
  68. 68. Outcomes Visibility Ownership Way  of  Working Changing  Conversations
  69. 69. www.suncorpgroupcareers.com.au
  70. 70. Thank you AWS Summit 2016, Sydney
  71. 71. Migration
  72. 72. Migration  Approaches One  time  Migration Periodic  Migration 1 2 3 Ongoing  Replication
  73. 73. Migration  Considerations Data Size,  Types,  Character  sets,  transfer  mechanism Dependencies Connected  Apps,  DB  Links,  Network,  Acceptable   downtime 1 2 3 Native  tools Oracle  Datapump,  pg_dump,  mysqldump,  SQL   Server  Management  Studio
  74. 74. Start  your  first  migration  in  10  minutes  or  less Keep  your  apps  running  during  the  migration Replicate  within,  to,  or  from  Amazon  EC2  or  RDS Move  data  to  the  same  or  different  database  engine   AWS Database  Migration   Service Amazon Aurora
  75. 75. Customer Premises Application  Users Internet VPN Keep  your  Apps  Running  During  Data  Migration
  76. 76. Customer Premises Application  Users Internet VPN AWS  Database Migration  Service Start  a  Replication  Instance Keep  your  Apps  Running  During  Data  Migration
  77. 77. Customer Premises Application  Users Internet VPN AWS  Database Migration  Service Connect  to  Source  and  Target  Databases Keep  your  Apps  Running  During  Data  Migration
  78. 78. Customer Premises Application  Users Internet VPN AWS  Database Migration  Service Select  Tables,  Schemas  or  Databases Keep  your  Apps  Running  During  Data  Migration
  79. 79. Customer Premises Application  Users Internet VPN AWS  Database Migration  Service Let  the  AWS  Database  Migration  Service  create  tables,  load  data Keep  your  Apps  Running  During  Data  Migration
  80. 80. Customer Premises Application  Users Internet VPN AWS  Database Migration  Service Let  the  AWS  Database  Migration  Service  keep  them  in  sync Keep  your  Apps  Running  During  Data  Migration
  81. 81. Customer Premises Application  Users Internet VPN Switch  applications  over  to  the  target  at  your  convenience AWS  Database Migration  Service Keep  your  Apps  Running  During  Data  Migration
  82. 82. AWS  Schema  Conversion  Tool Features Oracle,  SQL  Server,  MySQL  and  PostgreSQL  conversion  to  MySQL/Aurora/MariaDB/PostgreSQL Database  Migration  Assessment  report  for  choosing  the  best  target  engine Code  browser  that  highlights  places  where  manual  edits  are  required The  AWS  Schema  Conversion  Tool  helps   automate  many  database  schema  and  code   conversion  tasks  when  migrating  between   database  engines.
  83. 83. SCT  helps  with  Converting  tables,  views,  and  code Sequences User  Defined  Types Synonyms Packages Stored  Procedures Functions Triggers Schemas Tables Indexes Views Amazon Aurora
  84. 84. SCT  can  tell  you  How  Hard  the  Migration  will  be 1.  Connect  SCT  to   Source  and  Target   databases. 2.  Run  Assessment   Report. 3.  Read  Executive   Summary. 4.  Follow  detailed   instructions.
  85. 85. Demo Migrating  Data  with  the AWS  Database  Migration  Service Marc  Teichtahl – Solution  Architect,  AWS
  86. 86. What  You’ve  Learnt  Today • How  to  run  fast  and  scalable  relational  database  workloads   using  Amazon  RDS • How  to  migrate  your  data  to  Amazon  RDS • Rethink  what  you  can  do  with  databases
  87. 87. AWS  Training  &  Certification Intro  Videos  &  Labs   Free  videos  and  labs  to   help  you  learn  to  work   with  30+  AWS  services   – in  minutes! Training  Classes In-­person  and  online   courses  to  build   technical  skills  – taught  by  accredited   AWS  instructors Online  Labs   Practice  working  with   AWS  services  in  live   environment  – Learn  how  related   services  work   together AWS  Certification Validate  technical   skills  and  expertise  – identify  qualified  IT   talent  or  show  you   are  AWS  cloud  ready Learn  more:  aws.amazon.com/training
  88. 88. Your  Training  Next  Steps: ü Visit  the  AWS  Training  &  Certification  pod  to  discuss  your   training  plan  &  AWS  Summit  training  offer ü Register  &  attend  AWS  instructor  led  training ü Get  Certified AWS  Certified?  Visit  the  AWS  Summit  Certification  Lounge  to  pick  up  your  swag Learn  more:  aws.amazon.com/training
  89. 89. Thank  You

×