Cookpad’s Migration   Path to AWS       Cookpad Inc.      Genki Sugawara
About Me•  My work at Cookpad  o  Head of Infrastructure  o  Mission: Building and implementing Cookpad’s     infrastructu...
Contents•    About Cookpad•    Why AWS?•    AWS server and network configuration  •    Migration of service
About Cookpad
About Cookpad•  Recipe website used by over 15 million   people•  Over 1 million Recipes•  490 million monthly PVs•  Ruby ...
About Cookpad•  PC site  o  cookpad.com
About Cookpad•  Mobile site  o  m.cookpad.com
About Cookpad•  iPhone•  Android
PV 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:0010:0011:0012:0013:0014:0015:0016:0017:0018:00                         ...
About Cookpad          Variation in PVs across the yearPV     4月   5月   6月   7月   8月   9月 10月 11月 12月 1月   2月   3月
Why move to AWS?
Why AWS?1.  Speed2.  Distribution of Work3.  Cost
Why AWS?SpeedDistribution  of Work      o  Development speed  Cost
Why AWS?Speed               o  New servers currently require severalDistribution      weeks or more to prepare  of Work   ...
Why AWS?SpeedDistribution   o  Getting caught up in infrastructure  of Work                  issues causes large delays in...
Why AWS?SpeedDistribution   o  With AWS, it takes less than 10  of Work                  minutes to start up an instance. ...
Why AWS? SpeedDistribution  of Work      o  Ability to distribute work  Cost
Why AWS? Speed               Before AWSDistribution  of Work                          Request              Prep           ...
Why AWS? Speed               After AWSDistribution  of Work                              Prep  Cost               App     ...
Why AWS? Speed         o  Without AWS, distributing work is difficult:                   •  Need infrastructure skills/kno...
Why AWS?SpeedDistribution  of Work      o  EC2 seems a little too costly  Cost
Why AWS?               For example, here’s an unexpected “surprise” in               my EC2 monthly statement…SpeedDistrib...
Why AWS?               iDC:Charged according to greatest               bandwidthSpeedDistribution  of Work  Cost
Why AWS?               AWS:Charged by data transmitted               (Less cost for sites like Cookpad, which have peak an...
Why AWS?Speed               o  Charged by amount of data transmittedDistribution       •  Less costly when difference betw...
Server & Network  Configuration
Server & Network               Configuration             Current Network Network Security   DNS   AMIMonitoringRedundancy ...
Server & Network               Configuration               o  Simple 3-layer structure Network               o  Networks a...
Server & Network               Configuration             EC2’s Network Network Security   DNS   AMIMonitoringRedundancy  M...
Server & Network               Configuration               o  All servers located in same segment Network               o ...
Server & Network               Configuration Network Security   DNS               o  Two types of security groups set for ...
Server & Network               Configuration             Security group organization/structure Network Security   DNS   AM...
Server & Network               Configuration Network Security      o  Basic allows for mutual communication               ...
Server & Network               Configuration             Security group organization/structure Network Security   DNS   AM...
Server & Network               Configuration Network Security   DNS         o  Security groups for each role   AMI        ...
Server & Network               Configuration               o  Enable access from App groups to DB Network          groups ...
Server & Network               Configuration               o  Allows queries from Basic to DNS Network Security   DNS   AM...
Server & Network               Configuration Network       o  IP address are not specified for general                  ac...
Server & Network               Configuration Network Security               o  With EC2, internal IP addresses   DNS      ...
Server & Network               Configuration               o  DNS is organized into a 2-part Active-Active Network        ...
Server & Network               Configuration               o  DNS obtains name tag information Network          and config...
Server & Network               Configuration               o  resolv.conf is periodically reset by cron Network           ...
Server & Network               Configuration               o  Cron requests DNS’s Public DNS Network          Name(Public ...
Server & Network               Configuration               o  DNS’s internal IP is acquired as the IP Network          add...
Server & Network               Configuration               o  Acquired internal IP is written into resolv.conf Network    ...
Server & Network               Configuration Network Security   DNS         o  Clean installation of CentOS5.5   AMI      ...
Server & Network               Configuration               o  AMI for each role is created from the base Network          ...
Server & Network               Configuration Network Security   DNS         o  System network health monitoring   AMI     ...
Server & Network               Configuration               o  Nagios monitors server health status Network       o  Munin ...
Server & Network               Configuration               o  Started instances are automatically Network          monitor...
Server & Network               Configuration Network Security   DNS               o  Increasing availability   AMI        ...
Server & Network               Configuration              Mutual monitoring using Elastic IP Network                o  Use...
Server & Network               Configuration               o  Monitor public DNS name of each Network          elastic IP ...
Server & Network               Configuration               o  Health check is not performed if the Network          return...
Server & Network               Configuration               o  If the master health check fails, then Network          the ...
Server & Network               Configuration              Restoration from AMI using Nagios Network                o  When...
Server & Network               Configuration Network Security               o  Mutual monitoring using Elastic IP   DNS   ...
Server & Network               Configuration Network Security   DNS               o  Downtime is longer compared to   AMI ...
Server & Network               Configuration Network Security   DNS                           Data   AMI                (D...
Server & Network               Configuration               o  EC2 used only for Slaves Network               o  Data in EB...
Server & Network               Configuration               o  New slave created from snapshots Network Security   DNS     ...
Server & Network               Configuration               o  Data created from snapshot has same Network          replica...
Service Migration
Service MigrationiDC & EC2 Hybrid                   Internet
Service Migrationo  Service access is divided up between EC2 & iDC   using round robino  Read from DB comes from EC2o  Wri...
Service MigrationMoving the master DB to EC2                  Internet
Service Migrationo  The master DB is moved to EC2o  Before the move, iDC access is gradually stoppedo  Finally, iDC is com...
Thank you!
Upcoming SlideShare
Loading in …5
×

Cookpad AWS Seminar

4,831 views

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,831
On SlideShare
0
From Embeds
0
Number of Embeds
3,060
Actions
Shares
0
Downloads
16
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Cookpad AWS Seminar

  1. 1. Cookpad’s Migration Path to AWS Cookpad Inc. Genki Sugawara
  2. 2. About Me•  My work at Cookpad o  Head of Infrastructure o  Mission: Building and implementing Cookpad’s infrastructure, always working to improve speed, scalability, availability, back up, and security.•  Open source work o  Development of AWS tools •  elasticfox-ec2tag, IAM Fox, R53 Fox o  Ruby Library Development •  Zipruby, libarchive, rua, etc.
  3. 3. Contents•  About Cookpad•  Why AWS?•  AWS server and network configuration  •  Migration of service
  4. 4. About Cookpad
  5. 5. About Cookpad•  Recipe website used by over 15 million people•  Over 1 million Recipes•  490 million monthly PVs•  Ruby on Rails + MySQL
  6. 6. About Cookpad•  PC site o  cookpad.com
  7. 7. About Cookpad•  Mobile site o  m.cookpad.com
  8. 8. About Cookpad•  iPhone•  Android
  9. 9. PV 0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:0010:0011:0012:0013:0014:0015:0016:0017:0018:00 About Cookpad19:0020:00 PV variation during a single day21:0022:0023:00
  10. 10. About Cookpad Variation in PVs across the yearPV 4月 5月 6月 7月 8月 9月 10月 11月 12月 1月 2月 3月
  11. 11. Why move to AWS?
  12. 12. Why AWS?1.  Speed2.  Distribution of Work3.  Cost
  13. 13. Why AWS?SpeedDistribution of Work o  Development speed Cost
  14. 14. Why AWS?Speed o  New servers currently require severalDistribution weeks or more to prepare of Work o  We lack the some of the know-how to build our own servers Cost
  15. 15. Why AWS?SpeedDistribution o  Getting caught up in infrastructure of Work issues causes large delays in releases Cost
  16. 16. Why AWS?SpeedDistribution o  With AWS, it takes less than 10 of Work minutes to start up an instance. Cost
  17. 17. Why AWS? SpeedDistribution of Work o  Ability to distribute work Cost
  18. 18. Why AWS? Speed Before AWSDistribution of Work Request Prep App Infra Engineer Engineer Cost
  19. 19. Why AWS? Speed After AWSDistribution of Work Prep Cost App Engineer
  20. 20. Why AWS? Speed o  Without AWS, distributing work is difficult: •  Need infrastructure skills/knowledge •  Problems with security & stabilityDistribution of Work o  With AWS, distribution of work is made possible •  Very little specialized skill needed Cost •  Security/stability issues can be solved by giving authority where needed
  21. 21. Why AWS?SpeedDistribution of Work o  EC2 seems a little too costly Cost
  22. 22. Why AWS? For example, here’s an unexpected “surprise” in my EC2 monthly statement…SpeedDistribution of Work Cost
  23. 23. Why AWS? iDC:Charged according to greatest bandwidthSpeedDistribution of Work Cost
  24. 24. Why AWS? AWS:Charged by data transmitted (Less cost for sites like Cookpad, which have peak andSpeed non-peak times)Distribution of Work Cost
  25. 25. Why AWS?Speed o  Charged by amount of data transmittedDistribution •  Less costly when difference between peak of Work & non-peak times is especially large. o  Do away with excess investment into servers Cost
  26. 26. Server & Network Configuration
  27. 27. Server & Network Configuration Current Network Network Security DNS AMIMonitoringRedundancy MySQL
  28. 28. Server & Network Configuration o  Simple 3-layer structure Network o  Networks are partitioned at each layer Security DNS AMIMonitoringRedundancy MySQL
  29. 29. Server & Network Configuration EC2’s Network Network Security DNS AMIMonitoringRedundancy MySQL
  30. 30. Server & Network Configuration o  All servers located in same segment Network o  Instead of partitioned networks, Security security groups are used DNS AMIMonitoringRedundancy MySQL
  31. 31. Server & Network Configuration Network Security DNS o  Two types of security groups set for AMI instancesMonitoring •  Basic •  Security groups for each roleRedundancy MySQL
  32. 32. Server & Network Configuration Security group organization/structure Network Security DNS AMIMonitoringRedundancy MySQL
  33. 33. Server & Network Configuration Network Security o  Basic allows for mutual communication between basic ports DNS •  ping(icmp) AMI •  httpMonitoring o  Allows access from specific security groups •  Health monitoring tools (Nagios, etc.)Redundancy •  Performance monitoring tools (Munin, MySQL etc.)
  34. 34. Server & Network Configuration Security group organization/structure Network Security DNS AMIMonitoringRedundancy MySQL
  35. 35. Server & Network Configuration Network Security DNS o  Security groups for each role AMI •  Enables communication between roles themselvesMonitoring •  Enables communication betweenRedundancy each role and basic. MySQL
  36. 36. Server & Network Configuration o  Enable access from App groups to DB Network groups Security DNS AMIMonitoringRedundancy MySQL
  37. 37. Server & Network Configuration o  Allows queries from Basic to DNS Network Security DNS AMIMonitoringRedundancy MySQL
  38. 38. Server & Network Configuration Network o  IP address are not specified for general access. Security o  One exception are roles accessed from DNS Elastic Load Balancing, in which AMI 10.0.0.0/8 access is allowed •  Cannot specify source IPMonitoring •  Cannot specify security groupRedundancy o  Start iptables on all servers MySQL •  Helps  eliminate  human  error
  39. 39. Server & Network Configuration Network Security o  With EC2, internal IP addresses DNS cannot be fixed AMI •  Internal IP addresses end upMonitoring changed with stops & reactivations o  Use Internal DNS to block out IPRedundancy addresses MySQL
  40. 40. Server & Network Configuration o  DNS is organized into a 2-part Active-Active Network configuration •  Each is assigned an Elastic IP Security o  Each server references DNS with resolv.conf DNS Server AMIMonitoringRedundancy MySQL Server
  41. 41. Server & Network Configuration o  DNS obtains name tag information Network and configures domain information Security Ex.) Name:dev DNS → dev.ap-northeast-1.compute.internal AMIMonitoringRedundancy MySQL
  42. 42. Server & Network Configuration o  resolv.conf is periodically reset by cron Network •  When internal IP address changes, resolv.conf is reset Security •  If one DNS server stops, it is removed DNS from resolv.conf AMI ServerMonitoringRedundancy MySQL
  43. 43. Server & Network Configuration o  Cron requests DNS’s Public DNS Network Name(Public DNS Name is fixed by Security Elastic IP assignment) DNS Request Public DNS Name AMIMonitoringRedundancy MySQL
  44. 44. Server & Network Configuration o  DNS’s internal IP is acquired as the IP Network address associated with the Public DNS Security Name DNS Acquire Public DNS Name AMIMonitoringRedundancy MySQL
  45. 45. Server & Network Configuration o  Acquired internal IP is written into resolv.conf Network o  If the request isn’t returned, then it is removed from resolv.conf Security DNS Write internal IP AMIMonitoringRedundancy MySQL
  46. 46. Server & Network Configuration Network Security DNS o  Clean installation of CentOS5.5 AMI o  Root Device = EBS o  Currently, a mix of 32bit and 64bit,Monitoring but will move to 64bit only in theRedundancy future. MySQL
  47. 47. Server & Network Configuration o  AMI for each role is created from the base Network AMI o  Each AMI is given its own version Security o  Also implement system management tools DNS such as Chef AMIMonitoringRedundancy MySQL
  48. 48. Server & Network Configuration Network Security DNS o  System network health monitoring AMI •  Nagios + nrpeMonitoring o  Performance monitoringRedundancy •  Munin MySQL
  49. 49. Server & Network Configuration o  Nagios monitors server health status Network o  Munin monitors and records server performance data (e.g. CPU usage, load Security average, etc.) DNS Server AMIMonitoring ServerRedundancy MySQL
  50. 50. Server & Network Configuration o  Started instances are automatically Network monitored by Nagios・Munin Security o  Each instance is given a tag so the DNS appropriate type of monitoring can be identified. AMIMonitoringRedundancy MySQL
  51. 51. Server & Network Configuration Network Security DNS o  Increasing availability AMI •  Mutual monitoring using Elastic IPMonitoring •  Restoration from AMI using NagiosRedundancy MySQL
  52. 52. Server & Network Configuration Mutual monitoring using Elastic IP Network o  Used in Nagios & LDAP redundancy Security DNS AMIMonitoringRedundancy MySQL
  53. 53. Server & Network Configuration o  Monitor public DNS name of each Network elastic IP Security DNS AMI Monitors Public DNS NameMonitoringRedundancy MySQL
  54. 54. Server & Network Configuration o  Health check is not performed if the Network returning internal IP address is of the server itself. Security o  If the address differs from the server, then DNS health check is carried out o  →Back up always performs health check for AMI master Back-up performsMonitoring master health checkRedundancy MySQL
  55. 55. Server & Network Configuration o  If the master health check fails, then Network the back-up assigns itself an elastic Security ID DNS o  Elastic IP is moved from the master to the back-up, and switched to AMI failoverMonitoring Elastic IP moved to back-upRedundancy MySQL
  56. 56. Server & Network Configuration Restoration from AMI using Nagios Network o  When Nagios fails its health check, it Security is restored from AMI DNS o  Used in Munin, etc. AMI MonitorMonitoringRedundancy Starts instance Server MySQL (new instance)
  57. 57. Server & Network Configuration Network Security o  Mutual monitoring using Elastic IP DNS •  Applied to the server that we most AMI want to minimize downtimeMonitoring o  Restoration from AMI using NagiosRedundancy •  Applied to server allowing 5〜~10 minutes downtime MySQL
  58. 58. Server & Network Configuration Network Security DNS o  Downtime is longer compared to AMI keepalived, etc.Monitoring o  Currently looking into redundancy using HeartbeatRedundancy MySQL
  59. 59. Server & Network Configuration Network Security DNS Data AMI (Daily)MonitoringRedundancy Data MySQL
  60. 60. Server & Network Configuration o  EC2 used only for Slaves Network o  Data in EBS Security o  Snapshots of data taken daily DNS AMIMonitoring DataRedundancy (Daily) MySQL Data
  61. 61. Server & Network Configuration o  New slave created from snapshots Network Security DNS Data (Daily) AMI RestorationMonitoring New DBRedundancy Data MySQL Start up
  62. 62. Server & Network Configuration o  Data created from snapshot has same Network replication position Security o  Simplification of slave failover DNS AMI Restore CreateMonitoring New DB New Data Data (EBS)Redundancy MySQL
  63. 63. Service Migration
  64. 64. Service MigrationiDC & EC2 Hybrid Internet
  65. 65. Service Migrationo  Service access is divided up between EC2 & iDC using round robino  Read from DB comes from EC2o  Write to DB takes place in iDC
  66. 66. Service MigrationMoving the master DB to EC2 Internet
  67. 67. Service Migrationo  The master DB is moved to EC2o  Before the move, iDC access is gradually stoppedo  Finally, iDC is completely removed.  
  68. 68. Thank you!

×