ITO/MARS Program: Learning Structure, Reusability and Real-time Modeling in Teams of Autonomous Robots Manuela Veloso, Principal Investigator Tucker Balch, Co-Investigator School of Computer Science, Carnegie Mellon University

Learning Structure, Reusability and Real-time Modeling in Teams of Autonomous Robots Impact : Enables control of large-scale robot teams (10s to 100s of robots) Improves effectiveness of robot teams in adversarial tasks Enables rapid adaptability in dynamic environments New Ideas: Reuse solved subproblems Model opponent agents Hierarchically distribute planning and communication Learn appropriate layers of abstraction for hierarchical control Year 1 Year 2 Year 3 Integration of COTS robot platform Completion of software prototype Demo on 5-10 robots Demo on 10-20 robot team

Impact :

Enables control of large-scale robot teams (10s to 100s of robots)

Improves effectiveness of robot teams in adversarial tasks

Enables rapid adaptability in dynamic environments

New Ideas:

Reuse solved subproblems

Model opponent agents

Hierarchically distribute planning and communication

Learn appropriate layers of abstraction for hierarchical control

Research Challenges Goal: effective control of multi-robot teams. Challenging environments: complex: state is not fully observable; uncertain: non-deterministic effects of action; dynamic: pre-planning not feasible; adversarial: agents work against team goals. Constraints on distributed teams of small robots: unreliable communication with limited range; limited computational resources.

Goal: effective control of multi-robot teams.

Challenging environments:

complex: state is not fully observable;

uncertain: non-deterministic effects of action;

dynamic: pre-planning not feasible;

adversarial: agents work against team goals.

Constraints on distributed teams of small robots:

unreliable communication with limited range;

limited computational resources.

Research Challenges World model: Incompleteness, nondeterminism, dynamics Action model: Multiple probabilistic effects Execution: Possibly partially observable Assessment: Positive rewards at “goal” states Organization: Single versus multiagent systems Environment: Static, cooperative, adversarial

World model: Incompleteness, nondeterminism, dynamics

Action model: Multiple probabilistic effects

Execution: Possibly partially observable

Assessment: Positive rewards at “goal” states

Organization: Single versus multiagent systems

Environment: Static, cooperative, adversarial

How to Address the Challenges: Integrate low-level and high-level control Reactive behaviors ensure quick response, deliberative planning addresses long-term goal achievement. Reuse solutions to solved sub-problems Know your opponent Adversary modeling enables a robot team to respond strategically to different scenarios. Distribute planning and communication Reduces single-point failures. Improves autonomy, stealth and reliability.

Integrate low-level and high-level control

Reactive behaviors ensure quick response, deliberative planning addresses long-term goal achievement.

Reuse solutions to solved sub-problems

Know your opponent

Adversary modeling enables a robot team to respond strategically to different scenarios.

Distribute planning and communication

Reduces single-point failures.

Improves autonomy, stealth and reliability.

Multi-Layered Control Architecture Bridge low-level behavior-based action and high-level goal-driven deliberative planning Issues under investigation: Combine pre-determined functional layers: layered learning Learn the most effective task-dependent functional layer decomposition: learning structure

Bridge low-level behavior-based action and high-level goal-driven deliberative planning

Issues under investigation:

Combine pre-determined functional layers: layered learning

Learn the most effective task-dependent functional layer decomposition: learning structure

Autonomous Behavior Recognition using HMMs Robots and information processing agents must be able to autonomously recognize the behavior of other robots The perceived signal is characterized in terms of behavior-relevant state features Hidden Markov Models (HMMs) are used to represent and recognize strategic behaviors Multiple HMMs capture different robot behaviors

Robots and information processing agents must be able to autonomously recognize the behavior of other robots

The perceived signal is characterized in terms of behavior-relevant state features

Hidden Markov Models (HMMs) are used to represent and recognize strategic behaviors

Multiple HMMs capture different robot behaviors

Autonomous Behavior Recognition using HMMs R: robot being observed O: the observing robot Goal: for O to infer R’s “strategic behavior” from what it perceives of R’s physical actions Approach: State the problem as a behavior membership decision.

R: robot being observed

O: the observing robot

Goal: for O to infer R’s “strategic behavior” from what it perceives of R’s physical actions

Approach: State the problem as a behavior membership decision.

Autonomous Behavior Recognition using HMMs Assumptions: R acts according to a known set of behaviors h(i). While acting in its environment, it reactively chooses a behavior that matches the current state. O has a model of the set of possible behaviors. O’s task is to select which h(I) R is performing. performing.

Assumptions:

R acts according to a known set of behaviors h(i). While acting in its environment, it reactively chooses a behavior that matches the current state.

O has a model of the set of possible behaviors.

O’s task is to select which h(I) R is performing. performing.

Autonomous Behavior Recognition using HMMs HMMs are suitable for this recognition process because internal state of the observed agent R is naturally “hidden.” Environmental state features provide observations that can be used in the HMMs.

HMMs are suitable for this recognition process because

internal state of the observed agent R is naturally “hidden.”

Environmental state features provide observations that can be used in the HMMs.

Hierarchical Communication, Planning and Learning

Multi-robot Team Organization Hierarchical, layered decomposition of teams, learning, planning and communication. Observation and information sharing/fusion at each level. Learning at the individual and unit level: effects of action; prediction of adversaries; reuse of sub-problem solutions.

Hierarchical, layered decomposition of teams, learning, planning and communication.

Observation and information sharing/fusion at each level.

Learning at the individual and unit level:

effects of action;

prediction of adversaries;

reuse of sub-problem

solutions.

Advantages of Distributed Planning and Communication Reduced dependence on centralized control (and single point failure). Robust team performance in the face of unreliable, limited-range communication. Local planning resources used to solve local problems without reliance on higher levels. High-level planning (and communication) only necessary when local planning fails.

Reduced dependence on centralized control (and single point failure).

Robust team performance in the face of unreliable, limited-range communication.

Local planning resources used to solve local problems without reliance on higher levels.

High-level planning (and communication) only necessary when local planning fails.

Milestones: Year 1 1 Sep 99 Robot platform selection 1 Jan 00 TeamBots simulation extensions to support platform Hardware and software integration (1 robot) 1 Mar 00 Reusable subproblems prototype in simulation Adversary modeling prototype in simulation 1 Jun 00 Hierarchical multi-robot architecture prototype in simulation Hardware and software integration (5 robots) 1 Sep 00 Demonstration of multi-robot architecture on 5 robots

1 Sep 99

Robot platform selection

1 Jan 00

TeamBots simulation extensions to support platform

Hardware and software integration (1 robot)

1 Mar 00

Reusable subproblems prototype in simulation

Adversary modeling prototype in simulation

1 Jun 00

Hierarchical multi-robot architecture prototype in simulation

Hardware and software integration (5 robots)

1 Sep 00

Demonstration of multi-robot architecture on 5 robots

Milestones: Year 2 1 Oct 00 Extend reusable subproblems to team applications Extend adversary modeling to team applications 1 Jan 01 Hardware platform extended to >10 robots 1 Mar 01 Reusable subproblems and adversary modeling integrated with hierarchical multi-robot architecture 1 Jun 01 Integrated test of full system on >10 robots

1 Oct 00

Extend reusable subproblems to team applications

Extend adversary modeling to team applications

1 Jan 01

Hardware platform extended to >10 robots

1 Mar 01

Reusable subproblems and adversary modeling integrated with hierarchical multi-robot architecture

1 Jun 01

Integrated test of full system on >10 robots