This document discusses using Amazon EC2 Spot Instances and Auto Scaling to build cost-effective, scalable applications. It provides examples of using Spot Fleet and Auto Scaling groups together to create worker fleets for batch processing jobs that scale based on job queue depth and scale down when not in use. It also discusses using Spot Instances on Amazon EMR for task nodes and HDFS instances to reduce costs for big data workloads. Finally, it shows an example of running stateless web servers on Spot Instances behind a load balancer for high availability at low cost.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS Webinar
https://amzn.to/JPWebinar https://amzn.to/JPArchive
EC2
2019-03-06
Amazon EC2
[AWS Black Belt Online Seminar]

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
/ EC2
EC2
EC2
AWS
• Amazon EC2 Auto Scaling
• AWS Support

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS Black Belt Online Seminar
•
•
① 吹き出しをクリック
② 質問を入力
③ Sendをクリック
Twitter
#awsblackbelt

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• 2019 3 6
AWS (http://aws.amazon.com/)
• AWS
AWS
•
• AWS does not offer binding price quotes. AWS pricing is publicly available and is subject to
change in accordance with the AWS Customer Agreement available at
http://aws.amazon.com/agreement/. Any pricing information included in this document is provided
only as an estimate of usage charges for AWS services based on certain information that you
have provided. Monthly charges will be based on your actual use of AWS services, and may vary
from the estimates provided.

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• ( ) ( )
•
• ( )
• ( )
• CloudWatch ( )
•
• ( ) :
• ( )
• ( )
• ( )
• ( ) ( )

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- Amazon EC2

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
EC2購入オプションをどう組み合わせるか？

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon EC2
• 9
•
-
HPC/
•
1,000
•
• API
•
•

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- 110
• (
)
•
• AWS
•
• 2017 8 26
vCPU 110
( )
•
• ( )

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- 110
• (
)
•
• AWS
•
• 2017 8 26
vCPU 110
( )
•
• ( )

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
•
• ( )
•
•

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS OSS,
AWS
CloudFormation
AWS
OpsWorks
Amazon
EMR
Amazon
ECS
AWS Data
Pipeline
AWS BatchAuto Scaling
AWS Thinkbox
Deadline

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- 2017 11

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(1)

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(2)

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(3)

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(4)

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(1)
•
•

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(2)
•
•
•

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(1)
•

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(2)
•
•
•

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
•
• AWS
2
• Amazon EC2
• ( )

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• 2
•
•
• CloudWatch Events

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
-
•
•
• http://169.254.169.254/latest/meta-data/
• https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/ec2-
instance-metadata.html
•
• http://169.254.169.254/latest/meta-data/spot/instance-action
•
• 5
• https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/spot-
interruptions.html

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
{
"version": "0",
"id": "12345678-1234-1234-1234-123456789012",
"detail-type": "EC2 Spot Instance Interruption Warning",
"source": "aws.ec2",
"account": "123456789012",
"time": "yyyy-mm-ddThh:mm:ssZ",
"region": "us-east-2",
"resources": ["arn:aws:ec2:us-east-2:123456789012:instance/i-1234567890abcdef0"],
"detail": {
"instance-id": "i-1234567890abcdef0",
"instance-action": "action"
}
}

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- CloudWatch Events

32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- CloudWatch Events

33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS
CLI
(aws ec2 ...) (EC2)
run-instances
(request-spot-
instances)
one-time (sync) /
persistent (sync)
request-spot-
fleet
/
vCPU
( )
request (async) /
maintain (async)
EC2
create-fleet /
vCPU
( )
instant (sync) /
request (async) /
maintain (async)
Auto Scaling
(New!)

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
m4.large Spot ASG Min: 1 Max: 10
m5.large Spot ASG Min: 1 Max: 10
c4.xlarge O-D ASG Min: 1 Max: 10
Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
-
ASG
ASG
-
ASG
ASG: Auto Scaling Group
O-D: On Demand Instance

36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
m4.large Spot Instances
m5.large Spot Instances
c4.xlarge On-Demand instances
Availability
Zone 1
Availability
Zone 2
Availability
Zone3
ASG
Single ASG
ASG: Auto Scaling Group
O-D: On Demand Instance
RI: Reserved Instance

37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS
CLI
(aws ec2 ...) (EC2)
run-instances
(request-spot-
instances)
one-time (sync) /
persistent (sync)
request-spot-
fleet
/
vCPU
( )
request (async) /
maintain (async)
EC2
create-fleet /
vCPU
( )
instant (sync) /
request (async) /
maintain (async)
Auto Scaling
(New!)

39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(1)

41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(2)
• Load balancing:
• Flexible workloads:

42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(3)
• vCPU
•

43. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(4)
•
vCPU
•
•

44. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(5)
• EC2
•

45. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(6)
•
•
•

46. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
EC2 Auto Scaling Group (1)
• EC2 EC2 Auto Scaling

47. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
EC2 Auto Scaling Group (2)

48. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
EC2 Auto Scaling Group (3)
•
•

49. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
EC2 Auto Scaling Group (4)
•
• 1.
• 2.
• 3.
•
• 12
•
2
•
30:70
•
• 2 +3
• 7

50. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
< > EC2 Auto Scaling Spot Fleet (1)
EC2 Auto Scaling Spot Fleet
AZ
[1]
1. Total target capacity
2. "Optional On-demand
portion
3.
[2]
1. Desired Capacity
2. "On-Demand base"
3. OD:Spot Ratio
ELB
Launch Template
$latest, $default
[1] https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/spot-fleet.html#on-demand-in-spot
[2] https://docs.aws.amazon.com/ja_jp/autoscaling/ec2/userguide/AutoScalingGroup.html#asg-purchase-options

51. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• (Termination Policy)
• (Termination
Protection)
•
•
(Entering Standby)
•
(Suspending Processes)
• (Rebalancing
activities)
< > EC2 Auto Scaling Spot Fleet (2) -
•
•
• ( )
( )

52. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
-

53. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
-

54. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• - Amazon EC2 - Amazon EC2 | AWS
https://aws.amazon.com/jp/ec2/spot/pricing/

55. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(1)
EC2

56. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(2)
AWS CLI: describe-spot-price-history
$ aws ec2 describe-spot-price-history --instance-types m3.medium
{
"SpotPriceHistory": [
{
"Timestamp": "2019-03-04T21:36:38.000Z",
"AvailabilityZone": "ap-northeast-1a",
"SpotPrice": "0.109600",
"ProductDescription": "SUSE Linux",
"InstanceType": "m3.medium"
},
{
"Timestamp": "2019-03-04T21:36:38.000Z",
"AvailabilityZone": "ap-northeast-1c",
"SpotPrice": "0.109600",
"ProductDescription": "SUSE Linux",
"InstanceType": "m3.medium"
},
:

57. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

58. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
On Demand Auto-
Scaling group
Output S3
bucket
Availability Zone A
Availability Zone B
Spot Fleet
Upload object
into input S3
bucket
Job SQS
Queue
Auto Scaling group and Spot fleet
scale up based on queue depth and
scale down based on CPU utilization
CW metrics
Workers will
check for jobs
in the queue
Workers will update Job status
(start time, SLA end time, etc) in
DynamoDB
Uploads to S3 will trigger
a Lamda function to put
jobs in SQS and
DynamoDB
EFSEC2 instance
worker fleet
Cost-effective Batch Processing with Amazon EC2 Spot |
AWS Compute Blog
https://aws.amazon.com/blogs/compute/cost-effective-
batch-processing-with-amazon-ec2-spot/

59. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS Batch ( )
JobQueue
submit-job
list-jobs
describe-jobs
cancel-jobs
terminate-jobs
Compute Environment
ECR/Docker
Registry
S3/EFS/NFS
Autoscaling
ECS
Docker Image
JobDefinitio
n
job
AWS Batch

60. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
ビッグデータ領域のリファレンスアーキテクチャ
Master
Node (OD
or RI)
S3
EMRFS
Master Fleet
Task
Node
(Spot)
Task
Node
(Spot)
Task
Node
(Spot)
Task Fleet
Instance
Store
EBS
Volume
Instance
Store
EBS
Volume
Instance
Store
EBS
Volume
Core Fleet
HDFS
Instance Fleet
Core Node
(OD or RI)
Core Node
(OD or RI)
Core Node
(OD or RI)
Amazon EMR

61. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
コンテナを活用したステートレスウェブアプリケーション
Stateless Web
Server (Spot)
Stateless Web
Server (Spot)
Session State Data
Stateless Web Server
(On-Demand)
Elastic Load
Balancing
Amazon EC2
Auto Scaling
ECS
container 1
Stateless Web Server
(On-Demand)
ECS
container 2
ECS
container 1
ECS
container 2
https://github.com/awslabs/ec2-spot-labs/tree/master/workshops/ec2-spot-fleet-web-app

62. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
< > ECS
• ECS
• ECS
(/etc/ecs/ecs.config)
• ECS AMI

63. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
< > ECS
( )
•
• Draining
• ( )SNS
•
→

64. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
< > ECS
• CloudFormation
•
•
•
Draining
• SNS E

65. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

66. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
4+1

67. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
4+1 -
• (
)
•
• AWS
•
• 2017 8 26
vCPU 110
( )
•
• ( )

68. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
4+1 -

69. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

70. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS Auto Scaling
2019 -
AWS Auto Scaling 2019 | http://blog.father.gedow.net/2019/02/05/aws-auto-scaling-2019/
2019

71. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
EC2 Spot Instances Workshops
https://ec2spotworkshops.com/

72. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
• ( ) ( )
•
• ( )
• ( )
• CloudWatch ( )
•
• ( ) :
• ( )
• ( )
• ( )
• ( ) ( )

73. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
• 9
•
•
•
• CloudWatch Events
•
• :
•
•
•
• ( )

74. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Q&A
AWS Japan Blog https://aws.amazon.com/jp/blogs/news/

75. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS AWS
https://amzn.to/JPArchive

76. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
[ ] AWS Innovate
https://amzn.to/AWSInnovateJP

77. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS Well-Architected 個別技術相談会
•
•

78. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS Webinar
https://amzn.to/JPWebinar https://amzn.to/JPArchive