The document discusses how to use Amazon EC2 Spot Instances to save up to 90% on big data and machine learning workloads. It explains that Spot Instances provide access to unused Amazon EC2 computing capacity at steep discounts compared to on-demand prices. It then provides guidance on how to best utilize Spot Instances for both training machine learning models and deploying trained models, including using checkpointing and Auto Scaling to handle potential interruptions of Spot Instance availability.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Save Up to 90% on Big Data and
Machine Learning Workloads with
Amazon EC2 Spot Instances
Deepthi Chelupati
Senior Product Manager, Amazon EC2
Amazon Web Services
C M P 3 7 5
Xiaoyi Chen
Senior Software Development Engineer,
Amazon EC2
Amazon Web Services

3. Big Data and machine learning applications
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is fueling machine learning applications
Big Data and machine learning stack
How to best utilize Amazon Elastic
Compute Cloud (Amazon EC2) Spot
Instances for machine learning
workloads?

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Big Data and machine learning applications
Autonomous
vehicles
Image recognition
search
Genomic
sequencing
Data analyticsRecommendation
engines

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Big Data and machine learning stack
Amazon ECS
GPU CPU
AWS Batch
AWS platforms Third-party platforms
Infrastructure Frameworks
Apache
MXNet
on AWS
TensorFlow

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is fueling machine learning applications?
ALGORITHMS
DIVERSE &
LARGE
DATASETS
COMPUTE
Amazon EC2

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What problem are we solving?
Increased innovation More experimentation
How to best utilize Amazon EC2 Spot Instances for machine learning
workloads and save up to 90%

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key components of machine learning workloads
• Build and train
• Deploy

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EC2 purchasing options

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why Amazon EC2
Unused Amazon EC2 capacity offered at steep discounts that AWS can reclaim with two
mins of notification
Up to 90% discount over
Amazon EC2 On-Demand
prices
Increase throughput up to 10x
while staying in budget
Launch through AWS services
or integrated third-parties

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Save up to 90% using Amazon EC2 Auto Scaling and
Amazon EC2 Fleet
Reduce cost Optimize performance Eliminate operational overhead
Amazon EC2
Auto Scaling

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building and training a machine learning model

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
You may
look like
this ...

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How to build and train a machine learning model using
Amazon EC2 Spot Instances?
• Checkpointing
• Most of frameworks support checkpointing.
Apache MXNet on AWS, TensorFlow, Keras,
PyTorch
• Use AWS Auto Scaling Mixed Instance group to
launch into another pool
• Check for 2-minute instance termination notice via
instance metadata or Amazon CloudWatch metrics
• Using stop-start to restart faster
Over 95% of the instances were
not interrupted in the last 3
months

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deploy the machine learning model using Amazon
EC2 Spot Instances
• Inference workloads are stateless, fault-tolerant making them
a great fit for Amazon EC2 Spot Instances
• Deploy your model using AWS Auto Scaling groups across
multiple Amazon EC2 instance types, multiple availability
zones and multiple purchase models
• Use multiple machines for these workloads
• Great fit for batch workloads

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How can you further optimize training using
Amazon EC2 Spot Instances?

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fast.ai beat Google by training ImageNet in 18
minutes …
Total cost to train the models Amazon EC2 Spot Instances was $40

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sample code to deploy prediction model on
Amazon EC2 Spot Instances using Amazon ECS cluster
{ "ipcMode": null, "executionRoleArn": null, "containerDefinitions":
[ { "dnsSearchDomains": null, "logConfiguration": null,
"entryPoint": [ "/usr/local/bin/dockerd-entrypoint.sh" ],
"portMappings": [ {
"hostPort": 80,
"protocol": "tcp", "containerPort": 8080 }],
"command": ["mxnet-model-server", "--start",
"--models","https://s3.amazonaws.com/model-server/models/squeezenet_v1.1/squeezenet_v1.1.model"],
"linuxParameters": null,"cpu": 4096, "environment": [],"ulimits": null, "dnsServers": null,
"mountPoints": [], "workingDirectory": null, "secrets": null,"dockerSecurityOptions": null,
"memory": 4096, "memoryReservation": null, "volumesFrom": [],
"image": "awsdeeplearningteam/mxnet-model-server:1.0.0-mxnet-cpu",

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
"disableNetworking": null, "interactive": true,
"healthCheck": { "retries": 5, "command": ["curl", "http://127.0.0.1:8080/ping"], "timeout": 60, "interval": 10,
"startPeriod": 20 }, "essential": true, "links": null, "hostname": null, "extraHosts": null, "pseudoTerminal": true,
"user": null, "readonlyRootFilesystem": null, "dockerLabels": null, "systemControls": null, "privileged": null,
"name": "mms-spot-demo-ecs" } ], "placementConstraints": [],
"memory": "7000", "taskRoleArn": null, "compatibilities": [ "EC2" ],
"taskDefinitionArn": "arn:aws:ecs:us-east-2:349748304635:task-definition/mms-spot-demo:18", "family": "mms-spot-
demo",
"requiresAttributes": [{"targetId": null, "targetType": null, "value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"},{ "targetId": null,
"targetType": null, "value": null, "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"},
{"targetId": null, "targetType": null,"value": null, "name": "ecs.capability.container-health-check" }], "pidMode":
null,
"requiresCompatibilities": [ "EC2" ], "networkMode": null, "cpu": "4096", "revision": 18, "status": "ACTIVE",
"volumes": []}

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Session repeats
29th November
Save up to 90% on Big Data and machine learning workloads
with Amazon EC2 Spot Instances
4:00 PM – 5:00 PM | Hotel Mirage, Las Vegas

24. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deepthi Chelupati & Xiaoyi Chen
chelupd@amazon.com
cxiaoyi@amazon.com

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.