AWS VPC distilled for MongoDB devOps
Upcoming SlideShare
Loading in...5
×
 

AWS VPC distilled for MongoDB devOps

on

  • 3,577 views

Notes about Amazon VPC, a canonical architecture and finally how to implement MongoDB replica sets. My blog http://goo.gl/0guF2 has the color pictures. And the file is at ...

Notes about Amazon VPC, a canonical architecture and finally how to implement MongoDB replica sets. My blog http://goo.gl/0guF2 has the color pictures. And the file is at http://doubleclix.files.wordpress.com/2012/10/vpc-distilled-04.pdf. For some reason, slideshare trims the colors.

Statistics

Views

Total Views
3,577
Views on SlideShare
3,160
Embed Views
417

Actions

Likes
2
Downloads
18
Comments
1

5 Embeds 417

http://www.scoop.it 410
https://www.linkedin.com 3
http://www.linkedin.com 2
http://webcache.googleusercontent.com 1
http://www.docshut.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • If you are looking to deploy MongoDB in a VPC MongoDirector is the best tool. Automated deployment, scale & management - http://blog.mongodirector.com/deploy-mongodb-in-an-amazon-virtual-private-cloud-vpc/
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

AWS VPC distilled for MongoDB devOps AWS VPC distilled for MongoDB devOps Presentation Transcript

  • AWS VPC Distilled- For MongoDB devOps Krishna Sankar @ksankar October 27,2012
  • Essential  DevOps Ref:  h4p://speakerdeck.com/u/dampier/p/rock-­‐‑solid-­‐‑mongo-­‐‑ops 2
  • AWS  VPC  Top  10 1.  Any  mature  AWS  infrastructure  should   use  VPC  (for  prod  &  dev  !)  2.  VPC  is  not  that  hard,  but  really  requires   devOps  skills  3.   cccc  4.  Designing  VPC  is  a  heist,  single-­‐handed  !  5.  VPC  gives  greater  control  &  flexibility  –   use  the  force  wisely  &  keep  your  designs   simple  
  • AWS  VPC  Top  10 6.  VPC  allows  one  to  design  multi-­‐layered   security  –  security  groups  at  the   application  layer  &  network  layer  ACLs  7.  VPC  gives  isolation  semantics  viz  private   subnet  vs.  public  subnet,  routing  via   internet  gateway,  NAT  et  al  8.  Incorporate  resilience,  knowing  that  VPC   can  span  availability  zones    9.  For  now,  inter  VPC  routing  &  VPC  designs   across  regions  are  not  that  easy  10. Plan  your  Reserved  Instances   o  vpc  &  non-­‐vpc  RIs  are  separate  &  not  changeable.  They  will   work,  but  capacity  is  not  guaranteed  i.e.  you  could  get  into   trouble  when  you  bounce  the  instances  
  • Canonical  VPC  Architecture 6 10.200.0.0/1 Dev VPC - bnet   Public  su 0/24   ubnet   0. ud   Private  s 0/24   10.200.7 Amazon  Clo NAT   P>   10.200.8 0. IGW   <Public  I rod VPC Availability Zone : us-west-2a P /16 bnet   10.100.0.0 bnet   AZ : us-west-2b IGW   Public  su 0/24     ubnet P rivate  su /24   0. Private  s 0/24   10.200.9 0.0 10.100.1 10.100.2 0. t-1a y Zon e : us-eas NAT   Availabilit o  Normally  your  instances  would  be   P>   ast-1b <Public  I AZ : us-e created  here   st-1c o  With  a  public  DNS    A Z : us-ea ubnet   o  And  a  host  name  viz:   Private  s 0/24   0. ubnet   10.100.4 ec2-­‐46-­‐137-­‐23-­‐217.eu-­‐ Private  s 0/24   10.100.3 0. west-­‐1.compute.amazonaws.com   o  Amazon  does  protect  it’s  cloud  from   attacks  et  al.  Still  not  fully  secure,  and   less  control  (for  example  cannot   reconfigure  security  groups)  
  • Canonical  VPC  Architecture 6 10.200.0.0/1 Dev VPC - bnet   Public  su 0/24   ubnet   0. ud   Private  s 0/24   10.200.7 Amazon  Clo NAT   P>   10.200.8 0. IGW   <Public  I rod VPC Availability Zone : us-west-2a P /16 bnet   10.100.0.0 bnet   AZ : us-west-2b IGW   Public  su 0/24     ubnet P rivate  su /24   0. Private  s 0/24   10.200.9 0.0 10.100.1 10.100.2 0. t-1a y Zon e : us-eas NAT   Availabilit o  VPC   P>   ast-1b <Public  I AZ : us-e o  Create  a  separate  VPC  for  each   st-1c function  –  usually  dev  &  prod  A Z : us-ea ubnet   o  AWS  has  regions  (US-­‐Virginia,  US-­‐ Private  s 0/24   0. California,  US-­‐Oregon,  EU-­‐Ireland,   ubnet   10.100.4 Private  s 0/24   AsiaPac-­‐Singapore,  AsiaPac-­‐Tokyo  &   0. 10.100.3 SouthAmerica-­‐Sao  Paulo   o  Each  region  has  2  or  more   availability  zones   o     A  VPC  can  span  Availability  Zones,   but  not  Regions  
  • Canonical  VPC  Architecture 6 10.200.0.0/1 Dev VPC - bnet   Public  su 0/24   ubnet   0. ud   Private  s 0/24   10.200.7 Amazon  Clo NAT   P>   10.200.8 0. IGW   <Public  I rod VPC Availability Zone : us-west-2a P /16 bnet   10.100.0.0 bnet   AZ : us-west-2b IGW   Public  su 0/24     ubnet P rivate  su /24   0. Private  s 0/24   10.200.9 0.0 10.100.1 10.100.2 0. t-1a y Zon e : us-eas NAT   Availabilit o  Subnets   P>   ast-1b <Public  I AZ : us-e o  Create  multiple  subnets  in  a  VPC   st-1c o  Subnets  cannot  span  availability  A Z : us-ea ubnet   zones  (or  regions)  –  so  create  (at   Private  s 0/24   0. ubnet   10.100.4 least)  one  subnet  per  availability   Private  s 0/24   10.100.3 0. zone   o  There  are  two  types  of  subnets   Public  subnet  &  Private  subnet   o  We  will  take  a  look  into  each  of  them   in  the  next  couple  of  slides  
  • Canonical  VPC  Architecture 6 10.200.0.0/1 Dev VPC - bnet   Public  su 0/24   ubnet   0. ud   Private  s 0/24   10.200.7 Amazon  Clo NAT   P>   10.200.8 0. IGW   <Public  I rod VPC Availability Zone : us-west-2a P /16 bnet   10.100.0.0 bnet   AZ : us-west-2b IGW   Public  su 0/24     ubnet P rivate  su /24   0. Private  s 0/24   10.200.9 0.0 10.100.1 10.100.2 0. t-1a y Zon e : us-eas NAT   Availabilit o  Public  Subnets  (1  of  3)   P>   ast-1b <Public  I AZ : us-e o  Public  subnets  are  for  instances  that   st-1c need  to  be  accessed  from  the  A Z : us-ea ubnet   Internet  –  usually  web  servers,   Private  s 0/24   0. ubnet   10.100.4 application  servers,  cache  servers   Private  s 0/24   10.100.3 0. and  ssh  bastions  belong  in  this   category   o  The  instances  in  the  public  subnet   communicate  with  the  external   world  via  an  Internet  Gateway  (igw)    
  • Canonical  VPC  Architecture 6 10.200.0.0/1 Dev VPC - bnet   Public  su 0/24   ubnet   0. ud   Private  s 0/24   10.200.7 Amazon  Clo NAT   P>   10.200.8 0. IGW   <Public  I rod VPC Availability Zone : us-west-2a P /16 bnet   10.100.0.0 bnet   AZ : us-west-2b IGW   Public  su 0/24     ubnet P rivate  su /24   0. Private  s 0/24   10.200.9 0.0 10.100.1 10.100.2 0. t-1a y Zon e : us-eas NAT   Availabilit o  Public  Subnets  (2  of  3)   P>   ast-1b <Public  I AZ : us-e o  The  security  groups  determine   st-1c which  ports  are  open  and  for  which  A Z : us-ea ubnet   hosts   Private  s 0/24   0. ubnet   10.100.4 o  Usually  web-­‐server-­‐group(80,443),   Private  s 0/24   10.100.3 0. app-­‐server-­‐group  &  ssh-­‐group(22)   are  the  two  major  port  groups   o  You  can  also  restrict  the  hosts  that   can  communicate  via  the  security   groups  
  • Canonical  VPC  Architecture 6 10.200.0.0/1 Dev VPC - bnet   Public  su 0/24   ubnet   0. ud   Private  s 0/24   10.200.7 Amazon  Clo NAT   P>   10.200.8 0. IGW   <Public  I rod VPC Availability Zone : us-west-2a P /16 bnet   10.100.0.0 bnet   AZ : us-west-2b IGW   Public  su 0/24     ubnet P rivate  su /24   0. Private  s 0/24   10.200.9 0.0 10.100.1 10.100.2 0. t-1a y Zon e : us-eas NAT   Availabilit o  Public  Subnets  (3  of  3)   P>   ast-1b <Public  I AZ : us-e o  Unlike  the  non-­‐vpc  instances,  the   st-1c instances  in  the  public  subnets  do  A Z : us-ea ubnet   not  have  a  public  IP;  nor  do  they   Private  s 0/24   0. ubnet   10.100.4 have  a  host  name  that  is  externally   Private  s 0/24   10.100.3 0. resolvable   o  So  you  need  to  allocate  an  elastic  IP   &  then  assign  it  to  the  instance  in   the  public  subnet  
  • Canonical  VPC  Architecture 6 10.200.0.0/1 Dev VPC - bnet   Public  su 0/24   ubnet   0. ud   Private  s 0/24   10.200.7 Amazon  Clo NAT   P>   10.200.8 0. IGW   <Public  I rod VPC Availability Zone : us-west-2a P /16 bnet   10.100.0.0 bnet   AZ : us-west-2b IGW   Public  su 0/24     ubnet P rivate  su /24   0. Private  s 0/24   10.200.9 0.0 10.100.1 10.100.2 0. t-1a y Zon e : us-eas NAT   Availabilit o  Private  Subnets   P>   ast-1b <Public  I AZ : us-e o  The  instances  in  the  private  subnet   st-1c cannot  be  accessed  directly  from  the  A Z : us-ea ubnet   internet   Private  s 0/24   0. ubnet   10.100.4 o  By  default,  all  the  instances  inside  a   Private  s 0/24   10.100.3 0. VPC  can  access  the  private  subnet   o  Usually  a  NAT  instance  would  be   created  and  then  the  instances  in  the   private  subnet  can  access  out  –  this  is   mainly  for  upgrades  &  downloads  
  • Canonical  VPC  Architecture 6 10.200.0.0/1 Dev VPC - bnet   Public  su 0/24   ubnet   0. ud   Private  s 0/24   10.200.7 Amazon  Clo NAT   P>   10.200.8 0. IGW   <Public  I rod VPC Availability Zone : us-west-2a P /16 bnet   10.100.0.0 bnet   AZ : us-west-2b IGW   Public  su 0/24     ubnet P rivate  su /24   0. Private  s 0/24   10.200.9 0.0 10.100.1 10.100.2 0. t-1a y Zone : us-eas o  VPC  subnetting  patterns   NAT   Availabilit o  Put  your  web/application  server,   P>   b <Public  I AZ : us-east-1 cache  &  ssh  gateways  in  public   st-1c subnets  A Z : us-ea ubnet   Private  s 0/24   o  Database  servers  should  be  in  the   0. ubnet   10.100.4 private  subnet   Private  s 0/24   0. 10.100.3 o  Control  access  via  security  groups   o  You  can  add  network  level  ACLs  for   one  more  layer  of  security  (in  case   of  misconfiguration  at  the  security   group  layer)  
  • Canonical  VPC  Architecture 6 10.200.0.0/1 Dev VPC - bnet   Public  su 0/24   ubnet   0. ud   Private  s 0/24   10.200.7 Amazon  Clo NAT   P>   10.200.8 0. IGW   <Public  I rod VPC Availability Zone : us-west-2a P /16 bnet   10.100.0.0 bnet   AZ : us-west-2b IGW   Public  su 0/24     ubnet P rivate  su /24   0. Private  s 0/24   10.200.9 0.0 10.100.1 10.100.2 0. t-1a y Zon e : us-eas NAT   Availabilit o  Topics  for  another  day   P>   ast-1b <Public  I AZ : us-e o  Scale  out  the  web  tier  with   st-1c multiple  public  subnets  across  A Z : us-ea ubnet   availability  zone     Private  s 0/24   0. ubnet   10.100.4 o  Dynamic  Scaling  with  AutoScaling   Private  s 0/24   10.100.3 0. o  Load  balancing  with  ELB  across   subnets  &  regions   o   Disaster  Recovery  Architectures   with  cross  region  &  cross-­‐cloud  
  • VPC  Pragmatics 1.  Amazon  has  (very)  detailed  documentation   o  I  really  like  the  users  guide  http://goo.gl/loUcI  (Good  work  guys)   o  Print  it,  read  it  &  annotate  it  with  notes  in  the  margin  !  2.  Multi-­‐layer  security  capable   o  Security  groups     o  Network  ACLs  are  fully  open  initially  3.  No  private  DNS  –  you  need  to  run  your  own   DNS   o  I  use  /etc/hosts  –  it  is  not  scalable,  but  works  fine.  For  example   MongoDB  replication  needs  to  resolve  the  hosts  in  a  replica  set   o  AWS  Feature  request  -­‐  Private  DNS  in  Ruote53    4.  Have  granular  security  groups  for  more  control   o  ssh-­‐group  that  opens  port  22   o  web-­‐server-­‐group  that  opens  80  &  443   o  app-­‐server-­‐group  –  as  needed   o  db-­‐group  that  opens  database  ports    -­‐  3306  for  mysql,  27017  for  mongo   et  al  
  • VPC  Pragmatics 5.  Have  a  scheme  &  assign  IP  addresses  to   the  instances  –  private  and  public   subnet   o  The  host  name  will  be  created  from  this  IP  address   o  For  example  if  you  assign  an  IP  10.100.23.67  to  an  instance,   it  will  have  a  host  name  ip-­‐10-­‐100-­‐23-­‐67  6.  Use  a  different  port  than  22  for  ssh.   o  A  decent  port  scan  will  let  the  world  know  what  your  ssh   port  is   o  While  it  doesn’t  guarantee  any  more  security,  it  will  be  a   quick  defense    against  script-­‐kiddies  
  • MongoDB  Replicasets  over  Amazon  VPC   q Goals  (1  of  2)   ud   Amazon  Clo o  Security  (Healthcare-­‐ grade)   C Prod VP 6 o  Max  resilience  against  data   .0/1 bnet   10.100.0 loss   Public  su 0/24   0. bnet   o  No  SPOF  (Single  Point  Of   10.100.1 P rivate  su /24   0.0 10.100.2 Failure)   t-1a o  Consistently  high  average   y Zon e : us-easAvailabilit throughput   ast-1b AZ : us-e o  Reasonable  resilience  &   ast-1c ubnet   AZ : us-e Private  s failure  separation(  Start   ubnet   0.0/24   Private  s 0/24   10.100.4 0. with  availability  zones  &   10.100.3 extend  to  region  as   needed)   o  Balance  cost,  latency,   availability  &  survivability  
  • MongoDB  Replicasets  over  Amazon  VPC   q Goals  (2  of  2)   ud   Amazon  Clo o  Recoverability   o  >  1  Replicaset   C Prod VP 6 o  Operations  efficiency   .0/1 bnet   10.100.0 o  Good  backup  against  all   Public  su 0/24   0. bnet   possible  failure  scenarios   10.100.1 P rivate  su /24   0.0 10.100.2 o  Good  snapshot  strategy  for   t-1a frequent  &  consistent   y Zon e : us-easAvailabilit snapshots   ast-1b AZ : us-e o  Recovery  scripts/processes   ast-1c ubnet   AZ : us-e Private  s o  Operational  finesse  –  e.g.   ubnet   0.0/24   Private  s 0/24   10.100.4 0. Zero  Downtime  upgrades   10.100.3
  • MongoDB  Replicasets  over  Amazon  VPC   o  Web  Servers,  Application   ud   servers  &  cache  servers  in  the   Amazon  Clo public  subnet   C Prod VP 6 .0/1 bnet   10.100.0 o  MongoDB  Primary  in  the   Public  su 0/24   0. bnet   10.100.1 P rivate  su /24   same  availability  zone,  but  in   0.0 10.100.2 a  private  subnet   e : us-eas t-1a o  m2.xlarge  instance  with   y ZonAvailabilit ast-1b enhanced  I/O;  EBS   AZ : us-e ast-1c ubnet   AZ : us-e Private  s ubnet   0.0/24   Private  s 0/24   10.100.4 o  2  Secondary  MongoDB  in   0. 10.100.3 private  subnet  in  two   different  availability  zones   o  m1.large  instances;  EBS  
  • MongoDB  Replicasets  over  Amazon  VPC   ud   Amazon  Clo C Prod VP 6 q  Backup  Strategy   .0/1 bnet   10.100.0 o  Snapshot  from  Secondary   Public  su 0/24   0. bnet   o  Hourly  backup  –  rotate   10.100.1 P rivate  su /24   0.0 10.100.2 every  24  hrs   t-1a o  Daily  backup  –  rotate  every   y Zon e : us-easAvailabilit month   ast-1b AZ : us-e o  Weekly  backup  –  rotate   ast-1c ubnet   AZ : us-e Private  s yearly.     ubnet   0.0/24   Private  s 0/24   10.100.4 10.100.3 0. o  Keep  a  copy  in  a   different  cloud  
  • VPC/MongoDB  Pragmatics 1.  DNS  for  replication   o  MongoDB  replication  needs  to  resolve  the  hosts  in  a  replica  set   o  I  use  /etc/hosts  –  it  is  not  scalable,  but  works  fine.   o  AWS  Feature  request  -­‐  Private  DNS  in  Ruote53    2.  10Gen  has  (very)  detailed  documentation  3.  And  MongoDB  replication  setup  is  easy  &   straightforward   o  Thanks  Guys    4.  Snapshots  are  important   1.  Against  program  errors   2.  Against  database  corruption   3.  Against  operations  mistakes   4.  Snapshot  scripts  could  be  a  little  trickier   o  I  will  update  after  a  few  days  of  experience