Slides that outline my thesis on end to end approaches for autonomous driving.

1. Thesis Seminar:
Imitation Learning and
Direct Perception for
Autonomous Driving
Rocky Liang
University of Waterloo CogDrive Lab
Supervisor: Dr. Dongpu Cao
1

2. Table of Contents
▪ Problem Definition
▪ Conditional Imitation Learning
▪ Direct Perception
▪ Curvature Based Dynamic Controller
2

3. Goal: To design a policy that can operate an autonomous system in a more
streamlined way than the traditional robotics approach, which is complex and difficult
to scale.
Problem Definition
Environment
Understandin
g
Decision
Making
Sensor Input
Control
Action
Actuation
Imitation
Learning
Policy
3

4. Conditional Imitation Learning
1. Record expert demonstrations of (state, action) pairs
2. Initialize model weights
3. Update weights by minimizing the model’s predicted action and real action
Model
State/
Observation
Context
Expert Action/
Ground Truth
4

5. Conditional Imitation Learning - Context
▪ Driving is a multimodal task: there
can be multiple correct actions in
any given location
▪ Without a context, predicting the
appropriate action is an under
constrained problem
5

6. Conditional Imitation Learning - Dataset
Dataset used: CARLA Imitation Learning
Dataset
Available contexts:
▪ Follow lane
▪ Left turn
▪ Right turn
▪ Straight
Anatomy of a datapoint
Context
Input (Observation & context) Label (action)
Steering, throttle, brake
Vx
6
Scene Description
▪ Suburban scenery
▪ Single lane roads
▪ Intersections
▪ Other vehicles & pedestrians

7. Conditional Imitation Learning - Model 1
Context as Input Model
7

8. Conditional Imitation Learning - Model 2
Branched Model
8

9. Conditional Imitation Learning - Limitations
▪ Low explainability
▪ Why did it take that specific action?
▪ Hard to debug
▪ Immutable behavior
▪ The behavior shown in the dataset is the behavior the trained model will
have
9

10. Direct Perception
Environment
Understandin
g
Decision
Making
Sensor Input
Control
Action
Actuation
Imitation
learning
Policy
Affordance
Prediction
Actuation
10

11. Direct Perception - Affordances
● Affordances are states that are key to the
robot’s operation
● Unlike mediated perception which seeks to
build a full reconstruction of the environment,
direct perception only extracts affordances
from sensors
● Affordances:
○ Lane deviation
■ Heading error & crosstrack error
○ Road curvature
○ Distance to car in front
● Scales across different vehicles with minimal
vehicle specific development
● Affordances are fed to a controller to drive the
car
11
Nonzero curvature
Zero curvature

12. Direct Perception - Model
Added lane change
contexts
12

13. Data Recorder
Python Clients
Weather Client
NPC Client
Recorder Client
CARLA Server
User Input
Saved
Data
Display
Window
Left Lane Change
Right Lane
Change
Left Turn Right Turn
13

14. Curvature Based Dynamic Controller
Lateral Dynamic Model
Source: Vehicle System Dynamics,
Khajepour
Loss Calculation Update Delta
14
1. Propagate lateral model and get curvature prediction
2. Calculate loss between predicted curvature and
affordance curvature
3. Minimize loss by updating control input using its
gradient

15. Curvature Based Dynamic Controller
Lateral
Dynamic Model
Vehicle states from
sensors
Control input
Predicted states
Error
Calculation
Iteratively solve
Solver Loop
15

16. Results
16

17. Summary & Contributions
▪ Developed context aware imitation learning models for vehicle control
▪ Developed direct perception model for vehicle control
▪ Context aware affordance predictor
▪ Curvature based dynamic controller
▪ Learning based but still vehicle agnostic
▪ Data recorder
▪ Able to record contextual driving data including lane change affordances
17

18. Future Work
More robust context modeling
▪ Currently, contexts are discrete categories
▪ Cannot navigate more complicated intersections
▪ Would like to represent context in a continuous way
▪ How to represent this information?
▪ How to pass it to the network?
Domain adaptation
▪ Fine tuning with real world dataset (simple, low generalizability)
▪ Learn common scene representation across multiple domains (tough, high
generalizability)
18

19. Key Sources
▪ End-to-end Driving via Conditional Imitation Learning, Codevilla et al.
▪ Agile Autonomous Driving using End-to-End Deep Imitation Learning, Pan et al.
▪ Imitation Learning for Vision-based Lane Keeping Assistance, Innocenti et al.
19