For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/11/vision-language-representations-for-robotics-a-presentation-from-the-university-of-pennsylvania/ Dinesh Jayaraman, Assistant Professor at the University of Pennsylvania, presents the “Vision-language Representations for Robotics” tutorial at the May 2023 Embedded Vision Summit. In what format can an AI system best present what it “sees” in a visual scene to help robots accomplish tasks? This question has been a long-standing challenge for computer scientists and robotics engineers. In this presentation, Jayaraman provides insights into cutting-edge techniques being used to help robots better understand their surroundings, learn new skills with minimal guidance and become more capable of performing complex tasks. Jayaraman discusses recent advances in unsupervised representation learning and explains how these approaches can be used to build visual representations that are appropriate for a controller that decides how the robot should act. In particular, he presents insights from his research group’s recent work on how to represent the constituent objects and entities in a visual scene, and how to combine vision and language in a way that permits effectively translating language-based task descriptions into images depicting the robot’s goals.

1. Vision-Language
Representations for
Robotics
Dinesh Jayaraman
Assistant Professor,
University of Pennsylvania
Funding agencies:

2. A Robot’s “Control Loop”
decision-making
observation
How should the robot represent the information in its visual
observations? 2
representation
?

3. representation encoder
What is a Good Visual Representation?
𝝓(⋅) 𝝓(𝐱)
Image 𝐱
“dog”
Current state-of-the-art for many computer vision
tasks involves learned representations that are:
pretrained without supervision! 3

4. representation encoder
What is a Good Visual Representation?
𝝓(⋅) 𝝓(𝐱)
Image 𝐱
“dog”
… for Recognition?
Current state-of-the-art for many computer vision
tasks involves learned representations that are:
pretrained without supervision! 4
Good representations organize information conveniently for the task.

5. Background: Contrastive Unsupervised Learning
What is to stop the representation from collapsing to 𝒛 𝒙 = 𝟎 ∀𝒙?
To prevent this, push different images to have different representations
Pull two views of the same image together in the representation
8

6. Background: Contrastive Unsupervised Learning
What is to stop the representation from collapsing to 𝒛 𝒙 = 𝟎 ∀𝒙?
To prevent this, push different images to have different representations
Pull two views of the same image together in the representation
6
Key Idea of Contrastive Learning:
Train representations from a collection of unlabeled images by
pulling together and pushing apart the right image pairs

7. What is a Good Visual Representation for
Robotics?
𝝓(⋅) 𝝓(𝐨)
Image 𝐨 𝝅(⋅)
“rotate
gripper
+4°”
𝒂 = 𝝅(𝝓(𝒐))
representation encoder action “policy”
7

8. Overview: The Reinforcement Learning (RL)
Formalism
Action
Agent’s objective: maximize the discounted sum of “reward” over time by
executing a good action sequence 𝑎1, 𝑎2, … ,
max
𝜋
𝑅 𝜋 = 𝔼 ෍
𝑡=0
∞
𝛾𝑡𝑟 𝑠𝑡, 𝑎𝑡, 𝑠𝑡+1
State 𝑠𝑡
Agent Environment
unknown
𝑠 ∈ 𝑆
𝑎 ∈ 𝐴
𝑃(𝑠′|𝑠, 𝑎)
𝑟(𝑠, 𝑎, 𝑠′)
States
Actions
Transition function
Task Reward function
8

9. Task-Conditioned “Universal” Value Functions
• Optimal Value Function of A State
𝑉∗
𝑠𝑜 = 𝔼 ෍
𝑡=0
∞
𝛾𝑡𝑟 𝑠𝑡, 𝑎𝑡, 𝑠𝑡+1
“How good is this state for completing the task 𝑔 (if acting optimally)”?
• 𝑉 Value functions are a useful abstraction:
• Can guide policy improvement such as through RL
• Well-known “Bellman equation” constraints connecting 𝑉 values at
consecutive steps, permitting easy dynamic programming-style learning.
• Don’t require known actions
𝑉∗
𝑠0; 𝑔 = 𝔼 ෍
𝑡=0
∞
𝛾𝑡
𝑟 𝑠𝑡, 𝑎𝑡, 𝑠𝑡+1; 𝑔
… Conditioned On A Task g [Schaul et al 2015]
9

10. Key Idea: Representations as Value Functions
𝝓(⋅) 𝝓(𝐨) 𝝅(⋅)
“rotate
gripper
+4°”
𝒂 = 𝝅(𝝓(𝒐))
Image 𝐨
representation encoder policy
10

11. Key Idea: Representations as Value Functions
𝑽∗(⋅ ; 𝒈)
0.8
𝑽∗(𝒐; 𝒈)
𝝓(⋅) 𝝓(𝐨)
Image 𝐨
“squeeze
the
brush
dry”
or
task specification 𝑔
11

12. Key Idea: Representations as Value Functions
𝑽∗(⋅ ; 𝒈) 𝑽∗(𝒐; 𝒈)
𝝓(⋅) 𝝓(𝐨)
Image 𝐨
Goal image 𝐠 𝝓(𝐠) 0.8
distance
Representation 𝝓(⋅) should be rich enough so that it easily expresses 𝑉∗
12

13. Key Idea: Representations as Value Functions
𝑽∗(⋅ ; 𝒈) 𝑽∗(𝒐; 𝒈)
𝝓(⋅) 𝝓(𝐨)
Image 𝐨
𝝓𝒍
(𝐠) 0.8
distance
“squeeze
the
brush
dry”
Goal language 𝐠 𝝓𝒍
(⋅)
13
Train 𝝓, 𝝓𝒍 through training
the value function:
𝑽∗
𝝓 𝒐 , 𝝓𝒍
𝒈 = 𝒅(𝝓 𝒐 ,𝝓𝒍
𝒈 )

14. Pre-Train on Pre-Recorded In-the-Wild Human
Videos
Human videos are goal-directed, and abundant!
• Treat the final frame of any video as the goal
• Reward function? r = 1 for last step of video, 0 elsewhere.
• Actions not available, but no problem: we only care for 𝑉∗(𝒔)
Ego4D dataset
(Grauman 2022)
EpicKitchens
(Damen 2021)
14

15. Offline RL Value Function Training Objective
Encourages 𝑉∗
⋅ ; 𝑔 = ||𝜙 ⋅ − 𝜙 𝑔 ||2 to become a
valid (“Bellman-consistent”) value function.
Pulls every frame o
preceding g to have high
𝑉∗
𝑜 ; 𝑔
Consecutive frames should be at different distances from the goal
– pushes frames apart
All frames leading up to the goal should be close to goal
– pulls frames together
Training representations as value functions with offline RL
generates a new control-aware contrastive learning objective! 15

16. Results: Image ↔ Language-Goal Distance 𝒅(𝝓 𝒐 ,𝝓𝒍 𝒈 )
16

17. Results: Image↔ Image-GoalDistance𝒅(𝝓 𝒐 ,𝝓𝒍 𝒈 )
On demo data, our representations predict smooth goal-conditioned V*
on human and robot videos.
17

18. What Can We Do With 𝝓(⋅) and 𝝓𝒍
(⋅)?
• Use as representations for robot learning:
• Training robot policies on image representation with:
• behavior cloning
• language-conditioned behavior cloning [Lynch ’20]
• Use as dense reward functions to guide reinforcement policy learning:
• 𝑅 𝑜, 𝑎, 𝑜′
; 𝑔 = 𝑉∗
𝑜′, 𝑔 − 𝑉∗
𝑜, 𝑔 = ||𝜙 𝑜′
− 𝜙 𝑔 ||2 − ||𝜙 𝑜
− 𝜙 𝑔 ||2
• offline RL (reward-weighted regression [Peters ’07]) for policy learning from noisy
demos
• online policy improvement with trajectory optimization and RL (natural policy
gradient [Kakade ’01]) 18

19. Quantitative Results Summary
Results: Real-World BC / Offline RL From 20 Demos
Results: Language-Conditioned Behavior Cloning
19

20. Noisy demos → Language Goal-Based Policies
20
Goal: “Place the pineapple in the pot”

21. Noisy demos → Image Goal-Based Policies
Close Drawer (100% success) Pick and Place Melon (100%)
Push Bottle (90%) Fold Towel (90%)
21

22. Takeaways
For a robot to make decisions about good actions, in what format should it
internally represent the images from its camera stream?
• Modern visual representations leverage deep neural networks self-supervised from
large unlabeled datasets of images, largely focus on visual recognition use cases.
• No explicitly robot-focused representations before the work presented here.
• Training representations as goal-conditioned “universal value functions”: a
powerful new way to learn control-aware vision, language, (and other?)
representations.
22

23. Resources
23
The work presented here is covered the following papers:
• Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, Dinesh
Jayaraman. LIV: Language-Image Representations and Rewards for
Robotic Control. ICML 2023.
• Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert
Bastani, Vikash Kumar, Amy Zhang. VIP: Towards Universal Visual Reward
and Representation via Value-Implicit Pre-Training. ICLR 2023.
For further reading on self-supervised representations:
• Balestriero et al, A Cookbook of Self-Supervised Learning. arXiv 2023.