Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

20190318 Amazon EC2 スポットインスタンス再入門

547 views

Published on

AWS公式オンラインセミナー: https://amzn.to/JPWebinar
過去資料: https://amzn.to/JPArchive

Published in: Technology
  • Be the first to comment

20190318 Amazon EC2 スポットインスタンス再入門

  1. 1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 1 ( ) EC2 2019-03-18 Amazon EC2
  2. 2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 2 • • • • • • • • • • •
  3. 3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 3
  4. 4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 https://www.itmedia.co.jp/enterprise/articles/1902/22/news006.html
  5. 5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 5 AI AWS (1/4) - ITmedia https://www.itmedia.co.jp/enterprise/articles/1902/22/news006.html
  6. 6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 6
  7. 7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 7
  8. 8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 8
  9. 9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 9
  10. 10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 10 ( ) ( 5 ) 2019-03-18 Amazon EC2
  11. 11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 11
  12. 12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 12
  13. 13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 13 • • • • • • • • • • •
  14. 14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 14
  15. 15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 15
  16. 16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 16
  17. 17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 17 • • • • • • •
  18. 18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 18 • • • • • • •
  19. 19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 19
  20. 20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 20 AWS CloudFormation AWS OpsWorks Amazon EMR Amazon ECS AWS Data Pipeline AWS BatchAuto Scaling AWS Thinkbox Deadline
  21. 21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 21 • • • • •
  22. 22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 22
  23. 23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 23
  24. 24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 24 • • • •
  25. 25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 25
  26. 26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 26 • •
  27. 27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 27 • • • •
  28. 28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 28
  29. 29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 29
  30. 30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 30
  31. 31. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 31 •
  32. 32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 32 • 1 • ( )
  33. 33. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 33
  34. 34. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 34 AWS CLI (aws ec2 ...) (EC2) run-instances (request-spot- instances) one-time (sync) / persistent (sync) request-spot- fleet / vCPU ( ) request (async) / maintain (async) EC2 create-fleet / vCPU ( ) instant (sync) / request (async) / maintain (async) Auto Scaling (New!)
  35. 35. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 35 AWS CLI (aws ec2 ...) (EC2) run-instances (request-spot- instances) one-time (sync) / persistent (sync) request-spot- fleet / vCPU ( ) request (async) / maintain (async) EC2 create-fleet / vCPU ( ) instant (sync) / request (async) / maintain (async) Auto Scaling (New!)
  36. 36. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 36 AWS CLI (aws ec2 ...) (EC2) run-instances (request-spot- instances) one-time (sync) / persistent (sync) request-spot- fleet / vCPU ( ) request (async) / maintain (async) EC2 create-fleet / vCPU ( ) instant (sync) / request (async) / maintain (async) Auto Scaling (New!)
  37. 37. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 37 AWS CLI (aws ec2 ...) (EC2) run-instances (request-spot- instances) one-time (sync) / persistent (sync) request-spot- fleet / vCPU ( ) request (async) / maintain (async) EC2 create-fleet / vCPU ( ) instant (sync) / request (async) / maintain (async) Auto Scaling (New!)
  38. 38. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 38 m4.large Spot ASG Min: 1 Max: 10 m5.large Spot ASG Min: 1 Max: 10 c4.xlarge O-D ASG Min: 1 Max: 10 Availability Zone 1 Availability Zone 2 Availability Zone 3 - ASG ASG - ASG ASG: Auto Scaling Group O-D: On Demand Instance
  39. 39. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 39 m4.large Spot Instances m5.large Spot Instances c4.xlarge On-Demand instances Availability Zone 1 Availability Zone 2 Availability Zone3 ASG Single ASG ASG: Auto Scaling Group O-D: On Demand Instance RI: Reserved Instance
  40. 40. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 40 • Auto Scaling Group
  41. 41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 41 EC2 Auto Scaling Spot Fleet AZ [1] 1. Total target capacity 2. "Optional On-demand portion 3. [2] 1. Desired Capacity 2. "On-Demand base" 3. OD:Spot Ratio ELB Launch Template $latest, $default [1] https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/spot-fleet.html#on-demand-in-spot [2] https://docs.aws.amazon.com/ja_jp/autoscaling/ec2/userguide/AutoScalingGroup.html#asg-purchase-options
  42. 42. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 42 • • • • • • • • •
  43. 43. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 43
  44. 44. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 44 On Demand Auto- Scaling group Output S3 bucket Availability Zone A Availability Zone B Spot Fleet Upload object into input S3 bucket Job SQS Queue Auto Scaling group and Spot fleet scale up based on queue depth and scale down based on CPU utilization CW metrics Workers will check for jobs in the queue Workers will update Job status (start time, SLA end time, etc) in DynamoDB Uploads to S3 will trigger a Lamda function to put jobs in SQS and DynamoDB EFSEC2 instance worker fleet Cost-effective Batch Processing with Amazon EC2 Spot | AWS Compute Blog https://aws.amazon.com/blogs/compute/cost-effective- batch-processing-with-amazon-ec2-spot/
  45. 45. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 45 JobQueue submit-job list-jobs describe-jobs cancel-jobs terminate-jobs Compute Environment ECR/Docker Registry S3/EFS/NFS Autoscaling ECS Docker Image JobDefinitio n job AWS Batch
  46. 46. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 46 ビッグデータ領域のリファレンスアーキテクチャ Master Node (OD or RI) S3 EMRFS Master Fleet Task Node (Spot) Task Node (Spot) Task Node (Spot) Task Fleet Instance Store EBS Volume Instance Store EBS Volume Instance Store EBS Volume Core Fleet HDFS Instance Fleet Core Node (OD or RI) Core Node (OD or RI) Core Node (OD or RI) Amazon EMR
  47. 47. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 47 コンテナを活用したステートレスウェブアプリケー ション Stateless Web Server (Spot) Stateless Web Server (Spot) Session State Data Stateless Web Server (On-Demand) Elastic Load Balancing Amazon EC2 Auto Scaling ECS container 1 Stateless Web Server (On-Demand) ECS container 2 ECS container 1 ECS container 2 https://github.com/awslabs/ec2-spot-labs/tree/master/workshops/ec2-spot-fleet-web-app
  48. 48. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 48 • • • • •
  49. 49. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 49
  50. 50. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 50
  51. 51. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 51
  52. 52. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 52 • • • • • • •
  53. 53. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 53
  54. 54. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 54 • • • • • • • • • • •
  55. 55. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 55 • • • • • • • • • • •
  56. 56. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 56

×