Get behind your keyboard for an immersive experience with AWS DeepRacer and reinforcement learning. In this workshop, developers with no prior machine-learning experience can acquire new skills and apply their knowledge in a fun and exciting way. Join a pit crew and build and train ML models that you can take to the track for a chance to climb the AWS DeepRacer League leaderboard. Start your engines—the race is on! Bring your laptop.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Get hands-on with AWS DeepRacer and compete in
the AWS DeepRacer League
Brien Blandford
Partner Solutions Architect
Amazon Web Services
A I M 2 0 6 - R

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Agenda
• AWS DeepRacer origin
• RL for the Sunday driver
• Virtual simulator
• Rubber meets the road
• Under the hood

3. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How can we put machine
learning
in the hands of all
developers? Literally

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
1/18 scale autonomous
race car
AWS DeepRacer:An exciting wayfor developers to get hands-on experience with
machine learning
Global Racing LeagueVirtual simulator, to train
and evaluate

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer League, race for prizes and glory
The world’s first global, autonomous racing league
www.deepracerleague.com
Keen on setting up a race in your company? Please contact us.

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer problem formulation
State

8. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reinforcement learning in the broader AI context
Reinforcement
learning
Supervised
learning
Unsupervised
learning

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Machine learning overview
Supervised Unsupervised Reinforcement

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reinforcement learning in the real world
Reward positive
behavior
Don’t reward
negative behavior The result!

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reinforcement learning terms
Agent Environment State
Action
EpisodeReward

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The reward function
The reward function incentivizes particular
behaviors and is at the core of reinforcement
learning

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The reward function in a race grid
S G = 2
GoalAgent

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Incentivizing centerline behavior
0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
S 2 2 2 2 2 2 G = 2
0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
8.6 9.5 8.5 7.5 6.3 5.0 3.5 1.9
S 10.4 9.4 8.2 6.9 5.4 3.8 G = 2
8.6 9.5 8.5 7.5 6.3 5.0 3.5 1.9
Discount per step
0.9

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How does learning happen? Value function
Policy function

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
RL algorithms: Vanilla policy gradient
* Image Source: Landscape image is CC0 1.0 public domain
J()New
weights
New
weights
0.4 ± 𝛿 0.3 ± 𝛿

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer neural network architecture
Output - actionInput - state (image)

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Method Supervised learning
How it works Expert driver controls a real world
car, that has a camera. Save the images from the
camera as inputs and corresponding driving actions
(speed and steering angle) as outputs. Train a
model.
Result Provide state(image) into model and receive
driving action
RL vs. other approaches for robotic racing
Method Reinforcement learning
How it works Virtual agent repeatedly interacts
with a simulated environment and logs experience
(image, action, new state, reward). Experience is
used to train a model, and new model is used to
get more experience.
Result Provide state(image) into model and
receive driving action

20. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Lab 0 – AWS DeepRacer service resource creation
Objective: Set up your account resources to get you to the races!
Time: 5 min.
1. Find the lab content here:
https://github.com/aws-samples/aws-deepracer-workshops/
2. Navigate to:
Workshops/2019-AWSSummits-AWSDeepRacerService/Lab0_Create_resources

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Cloud
AWS
DeepRacer
NAT gateway
VPC
AWS DeepRacer
Models
Simulation
video
Metrics
AWS DeepRacer simulator architecture

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer console diagram

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Programming your own reward function

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Track components
Track center
Track wall
Track surface, a.k.a. on-track
Field, a.k.a. off-track
Track boundaries

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Coordinate system and track waypoints
Outer boundary waypoints
Track center waypoints
Inner boundary waypoints
X
Y
Track width
Car direction

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Action space

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Hyper parameters control the training algorithm

29. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer League, race for prizes and glory
The world’s first global, autonomous racing league
www.deepracerleague.com

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Submit your model now to race in the Virtual Circuit!

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Lab 1 – AWS DeepRacer service
Objective: Build your first AWS DeepRacer RL model
Time: 50 min.
1. Find the lab content here:
https://github.com/aws-samples/aws-deepracer-workshops/
2. Navigate to:
Workshops/2019-AWSSummits-AWSDeepRacerService/Lab1

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer: Driven by reinforcement learning
Want to learn more?
Learn how to build a reinforcement learning model, and find tips and tricks about
how to tune those models to climb the League leaderboard in a digital training
course for reinforcement learning and
AWS DeepRacer
This 90-minute course is available at no cost, has six self-guided chapters, and
helps you prepare to compete in the AWS DeepRacer League
https://www.aws.training/learningobject/wbc?id=32143

34. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer car specifications
Car 18th scale 4WD with monster truck chassis
CPU Intel Atom Processor
Memory 4-GB RAM
Storage 32 GB (expandable)
Wi-Fi 802.11ac
Camera 4 MP camera with MJPEG
Drive battery 1,000 mAh lithium polymer
Compute battery 13,600 mAh USB-C
Sensors Integrated accelerometer and gyroscope
Ports 4x USB-A, 1x USB-C, 1x Micro-USB, 1x HDMI
Software Ubuntu OS 16.04.3 LTS, Intel OpenVINO
toolkit, ROS Kinetic

36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
ROS msg node
Stored file
ROS nodes
Web
server
publisher
Model
optimizer
Video M-
JPEG
Web server
video
Inference
results
Autonomous
drive
Control
node
Optimized
model
Media engine
Camera
Model
Inference
engine
Manual
drive
Navigation
node
Servo & Motor
AWS DeepRacer software architecture

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Simulation-to-real domain transfer
Sim-to-real challenge
Train model using simulated images, but the
race car using the images the car experiences
in the real world
Strategies
Environment control
Domain randomization
Modularity and abstraction

38. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.