Optimize your Machine Learning Workloads on AWS (July 2019)
•Download as PPTX, PDF•
2 likes•217 views
Talk at Floor 28, Tel Aviv. Infrastructure, tips to speed up training, hyperparameter optimization, model compilation, Amazon SageMaker Neo, cost optimization, Amazon Elastic Inference
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Optimize your Machine Learning Workloads on AWS (July 2019)
Similar to Optimize your Machine Learning Workloads on AWS (July 2019) (20)
More from Julien SIMON
More from Julien SIMON (17)
Recently uploaded
Recently uploaded (20)
Optimize your Machine Learning Workloads on AWS (July 2019)
- 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Optimize your Machine Learning workloads onAWS Julien Simon Global Evangelist, AI & Machine Learning, AWS @julsimon F l o o r 2 8 , T e l A v i v , J u l y 7 t h , 2 0 1 9
- 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda • Infrastructure • Training • Easy tips to speed up the training process • Automatic model tuning • Inference • Model compilation: Amazon SageMaker Neo • Cost optimization: Amazon Elastic Inference
- 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EC2P3dn https://aws.amazon.com/blogs/aws/new-ec2-p3dn-gpu-instances-with-100-gbps-networking-local-nvme-storage-for-faster-machine-learning-p3-price-reduction/ Reduce machine learning training time Better GPU utilization Support larger, more complex models K E Y F E A T U R E S 100Gbps of networking bandwidth 8 NVIDIATesla V100 GPUs 32GB of memory per GPU (2x more) 96 Intel Skylake vCPUs (50% more than P3) with AVX-512
- 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EC2C5n https://aws.amazon.com/blogs/aws/new-c5n-instances-with-100-gbps-networking/ Intel Xeon Platinum 8000 Up to 3.5GHz single core speed Up to 100Gbit networking Based on Nitro hypervisor for bare metal-like performance
- 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tips tospeed up training • Scale out with distributed training • Pick the best format for your dataset • Use protobuf instead of CSV or JSON • Pack samples into record-based files • TFRecord (Tensorflow) or RecordIO (MXNet) • Splitting in 100MB files looks like the sweet spot • Amazon SageMaker: protobuf-encoded RecordIO • Use Pipe Mode for large datasets • Stream directly from Amazon S3, don’t copy • Train on arbitrary large datasets • Monitor CPU/GPU usage in Amazon CloudWatch • Use the largest batch size that makes sense for your dataset • Multiply batch size by the number of GPUs on the instance
- 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Examples of hyperparameters Neural Networks Number of layers Hidden layer width Learning rate Embedding dimensions Dropout … XGBoost Tree depth Max leaf nodes Gamma Eta Lambda Alpha …
- 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tacticsto find theoptimal setof hyperparameters Finding the optimal set of hyper parameters 1. Manual Search (”I know what I’m doing”) 2. Grid Search (“X marks the spot”) Typically training hundreds of models Slow and expensive 3. Random Search (Bengio 2012)Spray and pray”) Works better and faster than Grid Search But… but… but… it’s random! 4. HyperparameterOptimization: use Machine Learning Training fewer models Gaussian Process Regression and BayesianOptimization
- 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Demo: HPO with Keras https://gitlab.com/juliensimon/dlnotebooks/tree/master/keras/04-fashion-mnist-sagemaker-advanced
- 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Inference 90% Training 10% Predictions drive complexity and cost inproduction
- 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Model optimization isextremelycomplex Other architectures
- 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AmazonSageMakerNeo Trainonce,runanywherewith2xtheperformance K E Y F E A T U R E S Integrated with Amazon EC2 and Amazon SageMaker Free of charge! Open-source runtime and compiler; 1/10th the size of original frameworks github.com/neo-ai
- 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Compiling ResNet-50 for the Raspberry Pi Configure the compilation job { "RoleArn":$ROLE_ARN, "InputConfig": { "S3Uri":"s3://jsimon-neo/model.tar.gz", "DataInputConfig": "{"data": [1, 3, 224, 224]}", "Framework": "MXNET" }, "OutputConfig": { "S3OutputLocation": "s3://jsimon-neo/", "TargetDevice": "rasp3b" }, "StoppingCondition": { "MaxRuntimeInSeconds": 300 } } Compile the model $ aws sagemaker create-compilation-job --cli-input-json file://config.json --compilation-job-name resnet50-mxnet-pi $ aws s3 cp s3://jsimon-neo/model- rasp3b.tar.gz . $ gtar tfz model-rasp3b.tar.gz compiled.params compiled_model.json compiled.so Predict with the compiled model from dlr import DLRModel model = DLRModel('resnet50', input_shape, output_shape, device) out = model.run(input_data)
- 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Demo: Neo on SageMaker
- 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Right-sizingyour inferenceinfrastructure • Statistical ML models, small DL models, and of course dev/test infrastructure: CPU instances (C5) deliver the best cost/performance ratio • Very large DL models • GPU instances (P2 or P3) should work best, especially if you need high throughput • If not, C5n could be a reasonable alternative • But what about everything in between? • Mid-sized models • NLP models • Low throughput, low latency workloads • «Too slow on CPU, too expensive on GPU » ?
- 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Are youmaking themostof yourGPU infrastructure?
- 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon ElasticInference Reducedeeplearninginferencecostsupto75% K E Y F E A T U R E S Integrated with Amazon EC2 and Amazon SageMaker Support forTensorFlow and Apache MXNet Single and mixed-precision operations
- 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Demo: Elastic Inference on Amazon SageMaker
- 23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Getting started http://aws.amazon.com/free https://ml.aws https://aws.amazon.com/sagemaker https://github.com/aws/sagemaker-python-sdk https://github.com/awslabs/amazon-sagemaker-examples https://medium.com/@julsimon https://gitlab.com/juliensimon/dlnotebooks
- 24. Merci! © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Julien Simon Global Evangelist, AI & Machine Learning, AWS @julsimon
Editor's Notes
- TODO: Tensorflow 2.0
- 1/ The new Amazon EC2 P3dn instance 2/ With four-times the networking bandwidth and twice the GPU memory of the largest P3 instance, P3dn is ideal for large scale distributed training. No one else has anything close. 3/ P3dn.24xlarge instances offer 96vCPUs of Intel Skylake processors to reduce preprocessing time of data required for machine learning training. 3/ The enhanced networking of the P3n instance allows GPUs to be used more efficiently in multi-node configurations so training jobs complete faster. 4/ Finally, the extra GPU memory allows developers to easily handle more advanced machine learning models such as holding and processing multiple batches of 4k images for image classification and object detection systems
- These performance and accuracy trade offs are felt most acutely at the edge. 1/ IoT applications are usually running on devices, out there in the real world. This means that the accuracy of models can be felt quickly, and immediately. Consumer IoT applications have a high expectation of accuracy - such as Alexa detecting the wake word reliably - the accuracy of that model really matters to the overall experience. In industrial IoT, devices are often responsible for monitoring and maintaining and core manufacturing processes, or safety. The accuracy of a model here is critical. 2/ Applications running on IoT devices at the edge are commonly very sensitive to latency; it’s part of the reason why customers are running the workload there in the first place, because they can’t afford the round trip to the cloud and back. So any increase in that latency can have a meaningful impact on the success of the device itself. 3/ IoT applications are often incredibly resource constrained, in a way which is much more acute than in the cloud. The devices are smaller, and have less memory and processing power, which is a real problem for machine learning models. 4/ In many cases, IoT applications need to run on very diverse hardware platforms, with a dizzying myriad of processor architectures. To get any sort of performance, developers have to optimize 5/ Finally, one of the key benefits of machine learning can get lost; the ability to continually improve the model. IoT applications are great data generators, and once that data is “ground-truthed”, it can be used to build more sophisticated models. However, if the effort to optimize those improved models for the constraints and diverse hardware at the edge is high, then it’s less likely to happen, and developers are leaving money on the table. A real missed opportunity. 6/ We don’t think that customers should have to choose between accuracy and performance. It’s a false choice, with a high cost. So I’m excited to announce a new feature of SageMaker…