The document describes an adversarial actor-critic method for solving task-and-motion planning problems using planning experience. The method learns stochastic operator policies that guide a planner to solve problems more efficiently by sampling actions based on the learned policies rather than uniformly. It formulates the problem of learning operator policies from planning experience data and proposes an adversarial training approach that combines actor-critic reinforcement learning with demonstrations from a planner's experience to learn robust policies in a data-efficient manner. Experiments show the method improves planning efficiency over a raw planner and learns more robust policies than other baselines.

1. Adversarial actor-critic method for
task-and-motion planning problems
using planning experience
Beomjoon Kim,
Leslie Pack Kaelbling,
Tomas Lozano-Perez
Massachusetts Institute of Technology

2. Class of problems: sequential manipulation

3. Class of problems: sequential manipulation
·Example problem:

4. Class of problems: sequential manipulation
·Example problem:

5. Action-space representation: high-level operators
·Pick(O,P,G)
O: object to be picked
P: robot base pose
G: grasp
Pick motion

6. Approaches
Plan
everything
Planning hard Learning hard

7. An approach for planning
·Random sampling + graph search
·At every node, sample a finite number
of actions
·Search with the sampled actions

8. Challenge 1: infinite branching factor
·Pick operator example
O: object to be picked
P: robot base pose
G: grasp
Pick motion

9. Challenge 1: infinite branching factor
·Pick operator example
O: object to be picked
P: robot base pose
G: grasp
Pick motion

10. Challenge 2: expensive edge evaluation
·Pick operator example
O: object to be picked
P: robot base pose
G: grasp
Pick motion
IF
PathExists(P)
IKExists(G,O)
THEN
Picked(O)

11. Challenge 2: expensive edge evaluation
·Pick operator example
O: object to be picked
P: robot base pose
G: grasp
Pick motion
IF
PathExists(P)
IKExists(G,O)
THEN
Picked(O)

12. Challenge 2: expensive edge evaluation
·Pick operator example
O: object to be picked
P: robot base pose
G: grasp
Pick motion
IF
PathExists(P)
IKExists(G,O)
THEN
Picked(O)

13. Approaches
Plan everything
Learn a
complete solution
Planning hard Learning hard
Slow

14. Approaches
Plan everything
Learn a
complete solution
Planning hard Learning hard
Slow

15. Approaches
Plan everything
Learn a
complete solution
Planning hard Learning hard
Slow Not robust

16. Approaches
Plan everything
Learn a
complete solution
Planning hard Learning hard
Learn a stochastic policy
to guide planning

17. Operator policy learning problem formulation
·Assume that we are given a set of high-level operators
·Our objective is to learn a set of operator policies
·That maximizes

18. Operator policy learning problem formulation
·Assume that we are given a set of high-level operators
·Our objective is to learn a set of operator policies
·That maximizes

19. Operator policy learning problem formulation
·Assume that we are given a set of high-level operators
·Our objective is to learn a set of operator policies
·That maximizes
Continuous parameters
of the operator

20. Use an RL algorithm?
·Our problems are...

21. Using RL algorithms is difficult in our problems
·Our problems are...
·Expensive to generate data
·IK and motion planning calls for each edge

22. Using RL algorithms is difficult in our problems
·Our problems are...
·Expensive to generate data
·IK and motion planning calls for each edge
·Difficult to explore
·small feasible action regions - difficult to get meaningful reward
·long-term action dependencies

23. Solution: Use planning experience
· · ·
· · ·

24. Planning experience dataset
·Planning experience dataset

25. Two good things about Dpl
·Only feasible operator instances are included

26. Two good things about Dpl
·Only feasible operator instances are included
·Exploration is guided by the planning strategy

27. Using Dpl naively: Learning from demonstrations
·Apply supervised learning:
max
✓
X
si,i2Dpl
||⇡✓(si) ki||2

28. Using Dpl naively: Learning from demonstrations
·Apply supervised learning:
max
✓
X
si,i2Dpl
||⇡✓(si) ki||2
·Problem: Similar in Euclidean space != same
feasibility results

29. Better LfD: Adversarial training
·Discriminator training (Loss function learning):
·Generator training:
max
↵
X
si,i2Dpl
ˆQ↵(si, i) ˆQ↵(si, ⇡✓(si))

30. Adversarial training: the good
·The good: Loss function is learned

31. Adversarial training: the bad
·The bad: not all data points lead to a goal state
(suboptimal demonstrations)

32. Adversarial Monte-Carlo:
Actor-critic + suboptimal demonstrations
·Adversarial Q-function training:
·Policy training:

33. Adversarial Monte-Carlo:
Actor-critic + suboptimal demonstrations
·Adversarial Q-function training:
·Policy training:
Regression on Q

34. Adversarial Monte-Carlo:
Actor-critic + suboptimal demonstrations
·Adversarial Q-function training:
·Policy training:
Regression on Q Adversarial LfD

35. Experiments - hypotheses

36. Experiments - hypotheses
1. AdMon is more data-efficient than pure RL or pure LfD

37. Experiments - hypotheses
1. AdMon is more data-efficient than pure RL or pure LfD
2. Learning can improve planning efficiency

38. Experiments - hypotheses
1. AdMon is more data-efficient than pure RL or pure LfD
2. Learning can improve planning efficiency
3. Planning makes the learned policy robust

39. Domain description
·Problem instance:
·number of obstacles
·poses of obstacles
·shapes of obstacles

40. Domain description
·Reward function:
·-1 if obst not cleared
·1 if obst cleared
·0 if picked
·Problem instance:
·number of obstacles
·poses of obstacles
·shapes of obstacles

41. Executing learned policy by itself: Pure LfD
Pure LfD: GAIL

42. Executing learned policy by itself: Pure RL
Pure RL: DDPG, PPO

43. Policy by itself: AdMon is more data-efficient
Both RL + LfD: AdMon

44. Policy by itself: AdMon is more data-efficient
Both RL + LfD: AdMon
>10x improvement in data efficiency

45. Learning makes planning efficient
Raw planner

46. Learning makes planning efficient
AdMon + planning

47. Learning makes planning efficient
>2x improvement in planning efficiency

48. Planning makes the learned policy robust
AdMon w/o planner

49. Planning makes the learned policy robust
AdMon w/o planner
>2 times
improvement
in solution quality

50. Bad local optima without special treatment of Dpl

51. Thank you

52. Learned policies without planning

53. Learning improves planning efficiency

54. Learning improves planning efficiency
reached 95% optimal around 600s

55. Learning improves planning efficiency

56. Learning improves planning efficiency
reached 95% optimal around 1500s

57. Possible question - problem setup
·What about which object to pick?
·Assumed to be given
·How do you define rewards?
·1 if object cleared, -1 if not, 0 if simply picked
·What is the state space?
·It is represented by an approximate configuration space
obstacles that we call key-configurations, where we represent the
state with collisions at sparsely-yet-carefully chosen configurations

58. Possible criticisms
·Your planner is awful
·Yes indeed. But if you give me a good sampling-based planner,
which typically uses a uniform sampler, then as we have shown
in our experiments, we can do much better by learning the
stochastic policy and using it instead of the uniform sampler

59. Possible criticisms
·Does learning a policy always help in improving planning
efficiency?
·Short answer: No. It only helps you if your problem is
hard enough to the degree that uniform sampling would
suffer.
·This hardness can be measured by two properties:
expensive edge checking, and the small ratio of solution
region vs entire action space. These two characteristics are
what makes uniform policy a bad idea, because in
expectation, you need to try a lot of samples, and each
trial takes a lot of time

60. Possible questions
·Why does AdMon perform better?
·1. Local optima
·2. Better exploration
·Why does GAIL perform worse than other methods in the
second domain?
·There are too many suboptimal branches, more than the first
domain. It treats everything as optimal demonstrations, and so
the data is very noisy, which hurts the performance

61. Possible questions
·Your Q-function learning is biased
·Yes it is. If we had infinite amount of resources, then
we perhaps don't need to add the adversarial term. But
like in a typical statistical learning setup, we are trying
to make up for the limited amount of data by guiding
the training of the policy using adversarial bias.
·There is also a good reason to add the term, which
forces the state-action distribution to the ones that we've
encountered in the Dpl dataset.

62. Possible questions
·How come other learning approaches are worse than
uniform?
·Uniform policy, if sampled enough times, guarantees the
probabilistic completeness.
·Other algorithms have fallen into a local optima - ex. moving
the first few objects but not all - that prevents you from making
a progress

63. Relation to guided policy search
·Guided policy search:
·Requires a differentiable reward function because it needs to do
a trajectory optimization, whereas our domains have
discontinuous reward function (ex. with steps)
·In GPS, trajectory optimization allows you to perform
supervised learning using them to learn a policy, because
trajectory optimization gives you at least a locally optimal
solution. In our case, each branch in the tree is not guaranteed
to be even locally optimal. All we know is that each operator
instance is feasible, and they are, in fact, suboptimal
demonstrations where the degree of suboptimality varies a lot

64. Relation to other pure actor critic algorithms
·We use planning experience to explore the reachable
state space, by learning a generator for feasible operator
instances
·Other actor critic algorithms do not have this capability,
and they will waste a lot of exploration efforts in trying to
learn which actions are infeasible

65. Relation to GAIL
·GAIL is an inverse reinforcement learning algorithm that
assumes that the demonstrations that you have are optimal
·However, our planning experience dataset is not optimal -
only one branch leads you to the solution. So if we treat all
of the dataset as optimal demonstrations, then the dataset
will be very noisy
·Alternatively we can just use the branch that actually lead
to the goal. However, because there is a very few of them
(i.e 5 out of 50 in the conveyor belt domain), it is difficult
to learn anything meaningful

66. GAIL
·Performs by:
·Lear reward by maximizing the rewards of the
demonstration dataset, and minimizing the current policy's
actions
·Do DDPG gradient step to maximize the sum of the
learned rewards

67. Relation to AlphaX
·AlphaGo or AlphaZero is the system for solving two-player
games
·I think if you take the view point that AlphaX is solving the
two-player games, then any difference in the two-player game
vs a sequential mobile manipulation problem applies:
·Continuous vs discrete action spaces
·Expensive feasibility checks vs lookup in the rulebook,
·Key configuration representation vs. Image plane on the game
board
·Source of data: calling an external planner to solve the problem vs.
making RL episodes with the minimax trick,

68. AlphaX as a general learning algorithm
·If you take the view AlphaX as a general learning
algorithm for guiding search, then there are some
similarities in terms of the algorithm
·Learning from planning experience
·AlphaZero learn a network that predicts both value and a
stochastic policy. We also learn them both, although there is a
bias term in the value function
·Once the learning is done, then we used it in the heuristic
forward search, whereas AlphaZero uses MCTS. We can also
use it in MCTS.
·

69. ·One key similarity:
·The policy data, pi, for AlphaZero is generated by the result of search in the MCT
·In our case, all of the policy data is a result of feasibility checks