- Hindsight Experience Replay (HER) is an RL technique that allows agents to learn from unachieved goals as if they were achieved. This helps address the sparse reward problem in RL. - HER replays experiences with substituted goals, generating pseudo-transitions with new rewards. This increases the effective sample complexity for learning tasks with sparse rewards. - Experiments show HER improves performance on manipulation tasks like pushing, sliding, and pick-and-place, allowing policies to be learned for these tasks where vanilla RL fails.

1. Hindsight Experience Replay
OpenAI
Paper review
Presenter : Uijin Jung

2. Presenter
• Name : Uijin Jung
• Github : github.com/jinPrelude
• 16년 부산 원주민
• 경기도 3년 거주 억양 섞임

3. Contents
• Abstract
1. Introduction
2. Background
3. Hindsight Experience Replay
4. Experiments
5. Related work
6. Conclusion

4. 1. Introduction

5. 1. Introduction
• Reward engineering limits the applicability of RL in the
real world because it requires both RL expertise and
domain-specific knowledge.
• But dealing with sparse rewards is also one of the biggest
challenges in RL

6. • One ability humans have, unlike the current generation of
model-free RL algorithms, is to learn almost as much from
achieving an undesired outcome as from the desired one.
1. Introduction

10. :(

11. :(

12. :(

13. :(

14. :(
:)

15. :(
:)
Hindsight Experience Replay

16. 2. Background
• Reinforcement Learning
• Deep Q Learning (DQN)
• Deep Deterministic Policy Gradient (DDPG)
• Universal Value Function Approximators (UVFA)

17. 3. Hindsight Experience Replay
• Bit flipping environment

18. 3. Hindsight Experience Replay
• Bit flipping environment

19. 3. Hindsight Experience Replay
• Bit flipping environment
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}

20. 3. Hindsight Experience Replay
• Bit flipping environment
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}
n

21. 3. Hindsight Experience Replay
• Bit flipping environment
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}
n
A = {0, 1, …, n-1}

22. 3. Hindsight Experience Replay
• Bit flipping environment
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}
A = {0, 1, …, n-1}

23. 3. Hindsight Experience Replay
• Bit flipping environment
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}
A = {0, 1, …, n-1}

24. 3. Hindsight Experience Replay
• Bit flipping environment
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}
A = {0, 1, …, n-1}
g (goal)

25. 3. Hindsight Experience Replay
• Bit flipping environment
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}
A = {0, 1, …, n-1}
0 1 0 0 … 1 1 0 1
n
g (goal)

26. 3. Hindsight Experience Replay
• Bit flipping environment
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}
A = {0, 1, …, n-1}
0 1 0 0 … 1 1 0 1g (goal) =

27. =
3. Hindsight Experience Replay
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}
A = {0, 1, …, n-1}
0 1 0 0 … 1 1 0 1g (goal)
• Bit flipping environment

28. =
3. Hindsight Experience Replay
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}
A = {0, 1, …, n-1}
0 1 0 0 … 1 1 0 1g (goal)
• Bit flipping environment

29. 3. Hindsight Experience Replay
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}
• Bit flipping environment

30. 3. Hindsight Experience Replay
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}
40
• Bit flipping environment

31. 3. Hindsight Experience Replay
{0,1} {0,1} {0,1} {0,1} … {0,1} {0,1} {0,1} {0,1}
40
• Bit flipping environment

33. 0 1 1 g

34. 1 1 0 s0
0 1 1 g

35. 1 1 0 s0 …
0 1 1 g

36. 1 1 0 s0 … 1 0 0 sT
0 1 1 g

37. 1 1 0 s0 … 1 0 0 sT
≠
0 1 1 g

38. 1 1 0 s0 … 1 0 0 sT
≠
0 1 1 g
Episode reward : {-1, -1, …, -1, -1}

39. 1 1 0 s0 … 1 0 0 sT
≠
0 1 1 g
Episode reward : {-1, -1, …, -1, -1}

40. 1 1 0 s0 … 1 0 0 sT
≠
0 1 1 g
Episode reward : {-1, -1, …, -1, -1}
R
r0(s0, a0, r0, s1, g)
r1(s1, a1, r1, s2, g)
rt(st, at, rt, st+1, g)
rT−1(sT−1, aT−1, rT−1, sT, g)

41. 1 1 0 s0 … 1 0 0 sT
≠
0 1 1 g
Episode reward : {-1, -1, …, -1, -1}
R
r0(s0, a0, r0, s1, g)
r1(s1, a1, r1, s2, g)
rt(st, at, rt, st+1, g)
rT−1(sT−1, aT−1, rT−1, sT, g)

42. 1 1 0 s0 … 1 0 0 sT
≠
1 0 0 sT
Episode reward : {-1, -1, …, -1, -1}
R
r0(s0, a0, r0, s1, g)
r1(s1, a1, r1, s2, g)
rt(st, at, rt, st+1, g)
rT−1(sT−1, aT−1, rT−1, sT, g)

43. 1 1 0 s0 … 1 0 0 sT
1 0 0 sT
=
Episode reward : {-1, -1, …, -1, -1}
R
r0(s0, a0, r0, s1, g)
r1(s1, a1, r1, s2, g)
rt(st, at, rt, st+1, g)
rT−1(sT−1, aT−1, rT−1, sT, g)

44. 1 1 0 s0 … 1 0 0 sT
1 0 0 sT
=
Episode reward : {-1, -1, …, -1, 0}
R
r0(s0, a0, r0, s1, g)
r1(s1, a1, r1, s2, g)
rt(st, at, rt, st+1, g)
rT−1(sT−1, aT−1, rT−1, sT, g)

45. 1 1 0 s0 … 1 0 0 sT
1 0 0 sT
=
Episode reward : {-1, -1, …, -1, 0}
R
r0(s0, a0, r0, s1, g)
r1(s1, a1, r1, s2, g)
rt(st, at, rt, st+1, g)
rT−1(sT−1, aT−1, rT−1, sT, g)
R′
r0(s0, a0, r0, s1, sT)
r1(s1, a1, r1, s2, sT)
rt(st, at, rt, st+1, sT)
rT−1(sT−1, aT−1, rT−1, sT, sT)

46. R
r0(s0, a0, r0, s1, g)
r1(s1, a1, r1, s2, g)
rt(st, at, rt, st+1, g)
rT−1(sT−1, aT−1, rT−1, sT, g)
R′
r0(s0, a0, r0, s1, sT)
r1(s1, a1, r1, s2, sT)
rt(st, at, rt, st+1, sT)
rT−1(sT−1, aT−1, rT−1, sT, sT)

47. Memory
R
r0(s0, a0, r0, s1, g)
r1(s1, a1, r1, s2, g)
rt(st, at, rt, st+1, g)
rT−1(sT−1, aT−1, rT−1, sT, g)
R′
r0(s0, a0, r0, s1, sT)
r1(s1, a1, r1, s2, sT)
rt(st, at, rt, st+1, sT)
rT−1(sT−1, aT−1, rT−1, sT, sT)

48. Memory

49. Training
Memory

50. Training
Memory
Hindsight Experience Replay

60. 3. Hindsight Experience Replay

61. 4. Experiments
• Three different tasks : pushing, sliding, pick&place
• How we define MDPs
• Does HER improve performance?
• Does HER improve performance even if there is only one goal we care
about?
• How does HER interact with reward shaping?
• How many goals should we replay each trajectory with and how to
choose them?
• Deployment on a physical robot

62. • How many goals should we replay each trajectory
with and how to choose them?
• future — replay with k random states which come from the same episode as the
transition being replayed and were observed after it,
• episode — replay with k random states coming from the same episode as the transition
being replayed,
• random — replay with k random states encountered so far in the whole training
procedure.

63. 6. Conclusion
• We showed that HER allows training policies which push,
slide and pick-and-place objects with a robotic arm to the
specified positions while the vanilla RL algorithm fails to
solve these tasks.