Developing machine learning models requires a lot of effort which often needs to be repeated over time as data distributions change. In this session you will learn about some of the latest concepts in Automatic Machine Learning including how to apply them to speed up development and achieve robust models over time. You will learn how to run a custom labelling job using Amazon SageMaker Ground Truth to build a larger data set to fine-tune your model. You will also learn how to tune your model’s hyperparameters using Amazon SageMaker’s Automatic Model Tuning capabilities and understand the theory of how bayesian optimisation is automatically applied for more accurate results and faster tuning. This workshop requires a laptop and administrative access to your own AWS account.

1. S U M M I T
SYDNEY

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automatic Labelling and Model
Tuning with Amazon SageMaker
Eden Duthie
Specialist Solutions Architect
Amazon Web Services
Adam Lynch
Partner Solutions Architect
Amazon Web Services

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Agenda
• Introduction – 15 min
• Overview of Amazon SageMaker GroundTruth – 15 min
• Lab: Automatic Labelling with Amazon SageMaker GroundTruth – 45 min
• Working Lunch – 15 min
• Lab: Private Workforce Labelling – 30 min
• Overview of Model Tuning using Bayesian Optimisation – 15 min
• Lab: Automatic Model Tuning with Amazon SageMaker – 45 min

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS account for the labs
In order to complete this workshop you'll need an AWS Account with
admin access.
There are resources required by this workshop that are eligible for the
AWS free tier if your account is less than 12 months old.
We will supply some credits for other tasks.

5. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What data science tasks can be automated?
?

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Data science tasks
• Problem formulation
• Data collection and selection
• Feature engineering
• Model selection
• Algorithm selection
• Neural network architecture
• Deployment

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AutoML – definition and goals
AutoML aims to maximise the performace of machine learning programs
without human assistance and subject to a computational budget.
Core goals:
a) Good performance: good generalisation performance across various
input data and learning tasks can be achieved
b) No assistance from humans: configurations can be automatically done
for machine learning tools
c) High computational efficiency: the program can return a reasonable
output within a limited budget
Taking the Human out of Learning Applications:
A Survey on Automated Machine Learning
Quanming Yao et al. arXiv:
1810.13306v3 [cs.AI] 17 Jan 2019

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The challenges of AutoML adoption
Deep learning Human design Computational budget

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Some tough challenges in machine learning
Availability of labelled data The road not travelled Drift

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Promising approaches
Learning to learn Simulation Reinforcement learning

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
KERAS
AI Services
ML Services
ML Frameworks +
Infrastructure
Vision Speech Languages Chatbots Vertical
Amazon
SageMaker
Amazon SageMaker
Ground Truth
Amazon
Sagemaker RL
AWS Marketplace
for ML
Amazon
SageMaker Neo
AWS
Deepracer
AWS
Deeplens
Amazon
Personalize
Amazon
Forecast
Amazon
Textract
Amazon
Rekognition
Amazon
Lex
Amazon
Polly
Amazon
Transcribe
Amazon
Comprehend
Amazon
Translate
N E W
N E W
N E W
C5 C5n
P3 P3dn
Amazon Elastic Inference
AWS Inferentia
AWS Greengrass

13. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Data labeling and machine learning
Labelled data

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Data labelling is hard
Need to label large datasets
Requires humans to perform labelling
Becomes time consuming and costly
Difficult to achieve high accuracy for labels

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker: Build, train, and deploy ML
1
2
3
1
2
3

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Key features
Automatic labeling via
machine learning
Ready-made and
custom workflows
Label
management
Private and public
human workforce
Amazon SageMaker Ground Truth
Label machine learning training data easily and accurately

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker Ground Truth
How it works

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker Ground Truth
How it works

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker Ground Truth
How it works

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker Ground Truth
How it works

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker Ground Truth
How it works

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Step 1: Create a Ground Truth labelling job

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Step 2: Provide details for a labeling job

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Step 2: Provide details for a labeling job

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Step 2: Provide details for a labeling

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Labelling job is now running

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Labelling app: Human workers label the images

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Management Console: View labels for images

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Management Console: View labels

31. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

34. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Solving the following problem:
𝑚𝑎𝑥 𝑥 ∈ 𝐴 𝑓(𝑥)

36. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Defining ‘A’ using Amazon SageMaker
"ParameterRanges": {
"CategoricalParameterRanges":
[
{ "Name": "tree_method",
"Values": ["auto", "exact", "approx", "hist"]}
],
"ContinuousParameterRanges":
[
{ "Name": "eta", "MaxValue" : "0.5", "MinValue": "0" }
],
"IntegerParameterRanges":
[
{ "Name": "max_depth", "MaxValue": "10", "MinValue": "1", }
]
}

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Gaussian process regression

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Gaussian process posterior on the objective function

39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Acquisition function

40. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
On Amazon Sagemaker – configure the tuning job
tuning_job_config = {
"ParameterRanges": {…}
"ResourceLimits": { "MaxNumberOfTrainingJobs": 20, "MaxParallelTrainingJobs": 3 },
"Strategy": "Bayesian",
"HyperParameterTuningJobObjective": {
"MetricName": "validation:auc",
"Type": "Maximize"
}
}

41. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Launch the tuning job
tuning_job = smclient.create_hyper_parameter_tuning_job(
HyperParameterTuningJobName = "MyTuningJob“,
HyperParameterTuningJobConfig = tuning_job_config,
TrainingJobDefinition = training_job_definition)

42. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

43. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Eden Duthie
duthiee@amazon.com
Adam Lynch
atlynch@amazon.com