This document summarizes Maja Mataric's research on using communication to address problems in multi-robot learning. It discusses two key problems - hidden state where robots can't sense all information, and credit assignment where reward must be divided among agents. The research demonstrates communication can help by allowing robots to share sensory data to overcome hidden state, and share reinforcement signals to overcome credit assignment. Two experiments are described: box pushing by 2 robots, and social rule learning by 4 robots. In both cases, local broadcast communication helped the robots learn tasks more effectively by addressing the two problems.

1. Using Communication to Reduce Locality in Multi-Robot
Learning
By: Maja J. Mataric
Presentation By:
Derak Berreyesa
UNR, CS, 11/17/04

2.  Attempt to bridge the fields of machine
learning, robotics and distributed AI.
 Deals with two key problems:
 Hidden state
 Credit state

3. Hidden State
 Situated agents typically can’t sense all
information for completing the task and
learning to perform it efficiently.

4. Credit Assignment
 Arises because of reinforcement in a
distributed system is often provided at a global
level, and must somehow be divided over
multiple agents whose impact differs and
varies over time.

5. Solving the problems
 Apply communication as sensing and as
reinforcement, in each case through local
undirected broadcast.
 Demonstrated the idea on two multi-robot
learning experiments.

6. Two robots
 A tightly-coupled coordination task (box
pushing.)
 Communication for sharing sensory data to
overcome hidden state.
 Reinforcement data to overcome the credit
assignment.

7. Four Robots
 Loosely-coupled task, learning social rules,
(yielding and sharing information.)
 Uses Communication to bridge the gap
between global and local payoff.

8. In both cases
 The main goal is to increase the scope of
impact of a single agent.
 Clusters agents when they are tightly
interacting.
 Has the effect of making the system less
distributed and alleviates the hidden state and
credit assignment problems.

9. Communication as Sensing
 Sensors are in-accurate and un-reliable.
 Interaction between agents is very important.
 Communication can be used as a form of
sensing.
 Things that are hard to sense can be
communicated.
 Agents that broadcast their state learn better.
 There is still inaccuracy with sending the
messages.

10. Communication as Reinforcement
 It is hard for multi-agent systems to achieve
group-level coherence.
 Central controller maintains optimizations over
state space and sends commands to the
group.
 Information is usually not available and can’t
be completed in real time
 Communication poses a bottleneck.

11. Reinforcement (cont.)
 As multi-agent systems learn their behavior
changes resulting in inconsistencies.
 Credit assignment problem the level of the
individual because interaction with the other
agents delays the agents payoff.
 At the group level because local individual
behavior must be associated with global
outcomes.

12. Reinforcement (cont.) again.
 Communication as reinforcement enables
agents to locally share a reward in order to
overcome the credit assignment problem.

13. Communication for Shared Sensing
 2 robots pushing a box.
 Box is to heavy for one robot to push alone.
 Six-legged robots.
 Radio communication mechanisms.
 Whiskers that detect contact with the box.
 5 sensors that detect direction and distance
form the goal.
 Goal marked with a bright light.

14. Shared Sensing (cont.)
 Reinforcement learning framework
 Learning mapping between it’s sensors and
pre-programmed behaviors:
– Find-box
– Push-forward
– Push-left
– Push-right
– Stop
– Send-msg

15. Shared Sensing (cont.) again.
 Algorithm chose “best” action 75% of the time
and random action 25% of the time.
 Hidden state problem was solved by having the
two agents pool their sensory resources.
 Credit assignment is solved by each agent
telling the other what action to perform,
observe the outcome and share the reward or
punishment.

16. Shared Sensing (cont.) once more.
 Desired policy was learned by both robots in
over 85% of the trials.
 Its was learned on average in 7.3 minutes.
 There were about 40 trials.
 Each robot learned differently depending on
which side it was on.

17. Communication for Shared Reward
 4 robots.
 2 social rules:
– Yielding to each other
– Sharing information about the location of pucks.
 Infa-red and bump sensors.
 Radio communication
 Sensors don’t give information about robots
external state or behavior.

18. Shared Reward (cont.)
 Basis behaviors:
– Pickup
– Drop
– Home
– Wander
– Follow
– Send-msg

19. Shared Reward (cont.) again.
 Goal was to have robot improve it’s individual
collection of pucks while still yielding to another
and send messages to know when follow or
proceed to the location.
 “best” behavior would be 60% of time and
random 10%, it would follow 30% of the time.
 It is difficult for robots to learn social rules and
the credit assignment problem was a big
problem.

20. Shared Reward (cont.) once more.
 Locally sharing information was sufficient to enable the
group to learn social behaviors.
 The social policies of yielding and sharing were learned
by all robots in over 90% of the trials.
 They were learned on an average of 20 to 25 minutes.
 Unlike the first experiment sensory information was
individual and reinforcement was shared.
 This took care of the credit assignment problem.
 Radio broadcast communication proved to be robust.
 Lost data was mostly ignored, but did slow down
learning a little.

21. Conclusions
 Dealt with two problems of learning in multi
agent environments.
 Simple communication over local broadcast
can be used to address both problems.
 We had fun doing it!!!!