Learning robust rewards with adversarial inverse reinforcement learning

1. Learning Robust Rewards With Adversarial Inverse
Reinforcement Learning
Justin Fu, Katie Luo, Sergey Levine
University of California, Berkeley
+ /1 5:
#
8 0 0 6 2

2. Agenda
• ML
• R NA
G E (,) - (,) ) -
- - -
)
• I - - ( , ) -
• C
2

3. t
• 2 8 A 2 1 A ( C 2 2 C A
2 8
, A -2A 0 8 C
)
lns u s u s W
• nF
W dg W e c ow
hbi
• f aF
W owJ e cSI L
K
Wv yr R ow m hbi
3

4. Markov Decision Process
• (", $, %, &, ', ())
• "
• $
• ' ∈ [0,1]
• % " × $ × " ↦ ℝ
• & " ↦ ℝ：
• () " ↦ ℝ：
4

5. Inverse Reinforcement Learning
• p
R ] R b mx!"L
• rt Ma [ e ) 8 : ( ,+ 0
u #I E i oL u I xm
x
$" # yn a Z
• yn x ou L mx %L
5
max
"
)*~, log $" #
$" # =
1
2
exp(!"(#))
!"(#)：reward function
2：partition function

6. [] Inverse Reinforcement Learning
• ) + ) 1 1 + 6 6 ( 66
• , 6 +F a!Fg E R RIG
• R RI "($)Ge n a LM a F i[
C"($)F i[GdmE
n aF i[ o [
R RI F i[ n RM [
6
ℒ'()*'+ , = ./~1 log 56 $ = ./~1 76($) − log!
= ./~1 76($) − log ./~9
exp(76($)
"($)
ℒ=*>?@(' " = ./~9 76($) − ./~9[log("($))]

7. Generative Adversarial Nets [Goodfellow+, 14]
• ) ) , ( , , , , ) M
a M
• ) ) G ! M
"#$%$(') M
• , ( , , :D M
M )
) ) :D - )
•
min
,
max
/
0 1, 3 = 56 ~ 8#$%$(6) log 1(<) + 5> ~ 8?(>) log(1 − 1 < )
Discriminator true labels
for dataset
Discriminator false labels
for generated data
7

8. G
• 6
• A ,1 + 1 N
8
!" # =
1
&
exp *"(#)
1
&
exp *"(#) + .(#)
※GAIL[Ho & Ermon, 16]は, 報酬関数を隠に求めつつ方策を学習
[http://rll.berkeley.edu/deeprlcourse/d
ocs/inverserl.pdf ]
,1 + 1 DF

9. Adversarial Inverse Reinforcement Learning
)
• ]!, G # d Dc +4 1 4
D [
• $% d
• a +4 1 4 D d D
4 4 ( , )
9
&% !, # =
exp $%(!, #)
exp $%(!, #) + . # !)
̂01 !, # = log &% !, # − log 1 − &% !, #
= log
exp $%(!, #)
exp $%(!, #) + . # !)
− log
. # !)
exp $%(!, #) + . # !)
= $% !, # − log .(#|!)

10. IL
• R rt I IL
IL
a c Ds m
• ID
o rt! rt n
IL Di
10

11. bc Dli m
e d
d ( ) ) , ) ) ,
• bc bc n a
bc
• ! " + $("′) ! " $("′)D a
"′ " bc
11
"( ")
"* "+

12. ,
• ! " , $ " , % " , &(") ", "′,
•
,
12
! " + $ "+ = % " + &("′)
! " = % " + const
$ " = & " + const

13. eB o lg
e fa 9 N
• !(#) e#B o B
• ̂!(s) e#B o
• [ B e m ] n , (
• 9 Φ # ∶ ) ↦ ℝ
• ,-,/
∗
! 1B B m
• , ̂-,/
∗
̂! 1B B m
• + ) m
, ̂-,/
∗
#, 2 = ,-,/
∗
#, 2 − Φ(#)
13
,-
∗ #, 2 = ! # + 6789[softmaxA9,-
∗ #′, 2′ ]

14. a B c
• ) ( ,
• ̂" # = " # + &(s) &(s) D
• ( ,
14
̂" # = " # + const
̂"(#) = " # + &(s) = " # + ./01[Φ(#′)] − Φ(#)
d a c b
7 ̂8
∗
#, ; = " # + ./01[Φ(#′)] − Φ(#) + ./01[softmax@17 ̂8
∗
#A, ;A ]

15. Adversarial Inverse Reinforcement Learning
• D i
[D i i bd!D he
bdaN , + 9 , - 9
D ] gc - ( )
15
" ̂$,&
∗
!, ( = "$,&
∗
!, ( − Φ(!)

16. Adversarial Inverse Reinforcement Learning
)
• D! , ,
16
"#,% !, &, !′ =
exp ,#,%(!, &, !′)
exp ,#,%(!, &, !′) + 0(&|!)
,#,% !, &, !′ = 2# ! + 3ℎ% !5 − ℎ% !

17. Adversarial Inverse Reinforcement Learning
• D exp $∗ &, (, &′ = +, ( &)
• ,
D
17
ℎ∗ & = /∗ & + const
6∗ & = 7∗ & + const
$∗ &, (, &′ = 7∗ & + 8/∗ &9 − /∗ &
;(&, () /(&)
$∗ &, (, &′ = =∗(&, ()
6∗ & + 8ℎ∗ &′ − ℎ∗ & = 7∗ & + 8/∗ &′ − /∗ &

18. 18

19. • ( ) (
19
value
iteration
step
return for the policy

20. a
• a D 2
• ) ( ( a
20
e

21. e b a
• P d
D
21
) (
A
b i
) (
) (
d

22. •
original policy direct policy transfer AIRL
(re-optimal
state only reward)
[https://sites.google.com/view/adversarial-irl]
22

23. H imitation learning
• H G ]I [
]I E 6
• 1A L6 & , 1A
23

24. • .
v O n
• .
w n
e
• i
• r p ) ,( -
24
0 ≤ exp &',)(+, ,, +′) ≤ 1

25. • [Goodfellow+, 14] Generative Adversarial Nets, NIPS2014
• [Ziebart+, 08] Maximum Entropy Inverse Reinforcement Learning, AAAI2018
• [Ng+, 99] Policy invariance under reward transformations : Theory and
application to reward shaping, ICML1999
• [Finn+, 16] A Connection Between Generative Adversarial Networks, Inverse
Reinforcement Learning, and Energy-Based Models, NIPS2016
• [Finn+, 16] Guided Cost Learning: Deep Inverse Optimal Control via Policy
Optimization, ICML2016
• https://sites.google.com/view/adversarial-irl : Ant
25

26. Appendix
26

27. D A
( ) G A
27

28. D
N ( ) - A A D
! G
28

29. A
D !
29

30. A
30

31. Decomposability condition
• ! " + $ "% ! " $("′)
• ,
•
• "
) "′ − + "′ "′
31
!(") = ) "′ − + "′
- " + + "% = . " + )("′)
- " − . " = ) "% − + "%
!(") = - " − . "

32. Decomposability condition
• ! !′
• #(!) = const ,
32
, ! = - ! + const
/ ! = 0 ! + const
!1 !2
!3 !4
# !1 = 0 !2 − / !2 = 0 !3 − / !3 = 0 !4 − / !4
# !3 = 0 !1 − / !1 = 0 !2 − / !2
# !2 = 0 !1 − / !1 = 0 !3 − / !3 = 0 !4 − / !4
#(!4) = 0 !2 − / !2

33. reward shaping
• !(#) [#
• ̂!(s) [#
• [ N B9 B a , (
• !(#) + ) ]
' ̂(,*
∗
#, , = '(,*
∗
#, , − Φ(#)
'(
∗ #, , = ! # + 1234[softmax<4'(
∗ #′, ,′ ]
'(
∗ #, , − Φ(#) = ! # − Φ(#) + 1234[softmax<4'(
∗ #′, ,′ ]
'(
∗ #, , − Φ # = ! # + 123?[Φ(#′)] − Φ(#) + 123?[softmax<?'(
∗ #4, ,4 − Φ(#′)]
' ̂(
∗
#, , = ! # + 123?[Φ(#′)] − Φ(#) + 123?[softmax<?' ̂(
∗
#4, ,4 ]
( Φ # ∶ A ↦ ℝ )
33

34. , reward shaping
• ̂"($) = ' ̂(
∗
$, + − -./0[softmax90' ̂(
∗
$′, +′ ]
• ̂" $ = " $ + =(s)
• ̂" $ = " $ + const
34
' ̂(
∗
$, + = " $ + -./@[Φ($′)] − Φ($) + -./@[softmax9@' ̂(
∗
$0, +0 ]
̂"($) = " $ + -./@[Φ($′)] − Φ($)
=(s) = -./@[Φ($′)] − Φ($)

35. reward shaping
• D Φ "# ≠ const D D
• ( ) ) , ) ) ,
•
• "* : D
• ̂, " " D Φ "- = 1 Φ "0 = 2
35
"* "-
"0
2*
2-
345 Φ "# = 6
4#
7 "# ", 2 Φ("′)
2* ∶ 7 "- "*, 2* Φ "- + 7 "0 "*, 2* Φ "0
2* ∶ 7 "- "*, 2- Φ "- + 7 "0 "*, 2- Φ "0
後続状態に依存
=行動に依存

36. reward shaping
• , Φ "# = const
• , , Φ " = const
•
36
̂+ " = + " + const

37. AIRL Generator optimization
•
• CG
• , !"($) ,
37
maximize
,
-" ., 0 = 2, !"(., 0) − 2, log7(0|.)
maximize
9
ℒ;<=>?@A B = 29 !"($) − 29 logB($)

38. AIRL Discriminator optimization
•
•
38
!" #, % =
exp *"(#, %)
exp *"(#, %) + . % #)
max
"
ℒ 2 = 3
456
7
89 log !"(#4, %4) + 8=>
log(1 − !" #4, %4 )
= 3
456
7
89 log
exp *"(#4, %4)
exp *"(#4, %4) + . %4 #4)
+ 8=>
log
. %4 #4)
exp *"(#4, %4) + . %4 #4)
= 3
456
7
89 *"(#4, %4) + 8=A>
log . %4 #4) − 28CD>
log(exp *"(#4, %4) + . %4 #4))

39. AIRL Discriminator optimization
• !
39
"
"!
ℒ ! = %
&'(
)
*+
"
"!
,-(/&, 1&) −
"
"!
2*5678
log(exp ,-(/&, 1&) + @ 1& /&))
= %
&'(
)
*+
"
"!
,-(/&, 1&) − *A8
exp ,-(/&, 1&)
1
2
exp ,-(/&, 1&) +
1
2
@ 1& /&)
"
"!
,-(/&, 1&)

40. AIRL Discriminator optimization
• ) () ! = !#
! !# D ,
• ) ()
$
%
40
exp )∗(,, .) = !#
)∗ ,, . = log !# = 3∗ ,, .

41. N ]
• N +
Φ " ↦ ℝ
• ,, %(', )) → ',
• % ', ) ≠ %, ', ) ̂/
• /+ '′9 ) [ N
41
̂/ ', ), ', = / ', ), ', + 3Φ ', − Φ '
̂/ ', ) = / ', ) + 3Φ %(', )) − Φ '