• Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
6,439
On Slideshare
0
From Embeds
0
Number of Embeds
12

Actions

Shares
Downloads
18
Comments
0
Likes
10

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. What makes AWS invincible? Haruka Iwao, 2014/03/15
  • 2. Before talking about AWS
  • 3. About myself Haruka Iwao (@Yuryu) DevOps Engineer at FreakOut, Inc. Lived in Osaka, Tsukuba, Yokohama. Now in Tokyo. Playing FFXIV ARR
  • 4. Me Final Fantasy XIV ARR Status: Cleared the Coil Turn 5. Got my Allagan Weapon.
  • 5. Kindle Publishing Publishing Kindle books about the Linux Kernel Search “Yuryu Linux”
  • 6. About FreakOut, Inc. Not about freaking out :p Advertisement company Established in 2010 “Real-time Bidding”
  • 7. Real-time bidding SSP Supply-side Platform DSP Demand-side Platform DSP Demand-side Platform DSP Demand-side PlatformRequest a page Read an ad tag Call for bids DSP decides the best ad for the user and page
  • 8. Real-time bidding (2) SSP Supply-side Platform DSP Demand-side Platform DSP Demand-side Platform DSP Demand-side Platform Bid Auction Return the winning ad
  • 9. Real-time bidding (3) http://londoncreative.com/real-time-bidding-spending-to-significantly-increase/
  • 10. Our motto 50ms or die. Return a response within 50ms or lose an auction automatically. Latency matters. Literally.
  • 11. How we use AWS
  • 12. Our system at a glance http://aws.amazon.com/jp/solutions/case-studies/freakout/
  • 13. Mix of on-premise and AWS On-premise in Japan AWS in North America Starting small Scaling well No need to visit a DC
  • 14. Latency matters
  • 15. Latency matters Latency is important for our service 1ms = 1/50 of processing time
  • 16. Latency between servers Freedom to build an arbitrary network ... Gives you an arbitrary latency
  • 17. Longer latency in AWS  On-premise time=0.063 ms time=0.083 ms time=0.077 ms time=0.070 ms time=0.092 ms time=0.069 ms time=0.077 ms  AWS, extreme case time=1.88 ms time=1.96 ms time=2.60 ms time=3.72 ms time=2.46 ms time=1.05 ms time=2.37 ms
  • 18. Longer latency in AWS (2) Hard to see? Let’s make a graph...
  • 19. Longer latency, illustrated 0 0.5 1 1.5 2 2.5 On-premise AWS RTT(ms) RTT(ms)
  • 20. Longer latency in AWS (3) This is not always true Just an extreme case This applies to intra-AZ “Option” to group servers in near racks would be great
  • 21. Placement groups Placement groups are not enough Only available to cluster compute instances Guarantees bandwidth, not latency
  • 22. Possible workarounds Assume the latency Design your app accordingly Use persistent connections Put hot data on local Still, lower latency gives “extra” room
  • 23. Infrastructure as Code
  • 24. The “Awesome” Console
  • 25. ... So awesome to make mistakes easily...
  • 26. AWS is Programmable.
  • 27. Thou hast SDK. Python
  • 28. Thou hast CLI. CLI
  • 29. Thou hast CloudFormation. AWS CloudFormation
  • 30. SDK + CLI + CloudFormation You can “code” your infrastructure Infrastructure becomes “reproducible” and “reusable”
  • 31. Always use CLI Always use CLI to make changes “Review” the commands Less chance of “oops” But...
  • 32. CLI is hard to understand!
  • 33. VS aws ec2 run-instances --image-id ami-xxxxxxxx -- count 1 --instance-type t1.micro --key-name MyKeyPair --security-group-ids sg-xxxxxxxx --subnet-id subnet-xxxxxxxx
  • 34. Easy enough?
  • 35. No way...
  • 36. Record & Play “Record” instructions on the Web Console “Playback” them using CLI In other words...
  • 37. Converted to aws ec2 run-instances --image-id ami-xxxxxxxx -- count 1 --instance-type t1.micro --key-name MyKeyPair --security-group-ids sg-xxxxxxxx --subnet-id subnet-xxxxxxxx
  • 38. With “playback” You could review changes beforehand You could record changes and reuse them Easier than writing CLI commands by hand
  • 39. A very famous quote about “code”
  • 40. All your code are belong to test
  • 41. Testing is Important Every program has bugs “Infrastructure as Code” is no exception
  • 42. How do you test?How do you test?
  • 43. Bugs can be fatal A bug can destroy your whole system What if you accidentally Terminate an instance Set a wrong route table Delete RR from Route53
  • 44. “Sandbox” for testing VPC is (sometimes) not enough Test 100% bootstrap in a safe environment Register IAM accounts Add Route53 zones Set up S3 buckets, etc…
  • 45. Framework for testing Test-kitchen to test your Chef cookbooks Serverspec to test your server setups How do you verify your changes to AWS?
  • 46. Possible workarounds Use a separate account Maybe we need more environments in the future? Costs money CloudFormer converts environments to configuration
  • 47. Scenario #1 You add a new rule to your security group aws ec2 authorize-security-… You want to make sure a port is open or closed between particular hosts How?
  • 48. Workaround #1 Create a new VPC Apply the new rule Launch two instances Check connectivity
  • 49. Scenario #2 You set up Route53 Health Checks Now you want to test if it actually fails-over How?
  • 50. Workaround #2 Set up two ELBs / instances Stop instances registered to one ELB Query to R53 until it fails- over
  • 51. Need a solution! A “common language” to verify AWS configuration Want to run tests cheaper, quicker and safer Even the requirements are not yet clear…
  • 52. In the end of the presentation…
  • 53. What makes AWS invincible? Lower latency Giving options or hints to EC2 “Playback” feature Generate CLI commands using simple UI Testing methodology
  • 54. Thank you!