SlideShare a Scribd company logo
1 of 34
Download to read offline
Policy Gradient
to
Actor-Critic
박진우 (Curt Park)
RL Korea Bootcamp
2019. 10. 27
발표자 소개
•컴퓨터공학 학사 (2006 - 2014)
•SW developer at Smilegate (2013.11 - 2014.5)
•SW developer at Ericsson (2014.10 - 2017.01)
•Research Engineer at Medipixel (2018.11 - 2019.08)
•Research Engineer at J.Marple (2019.09 - )


3
걸어온 길
최근 활동
•연구주제
- 모델 기반 동적 제어 시스템
- 강화학습을 이용한 심혈관 시술용 가이드와이어 제어
•기타활동
- PG is all you need with 김경환, 김민철
- Rainbow is all you need with 김경환
- RL algorithms with 김경환, 김민철
- 모두를 위한 컨벡스 최적화 @풀잎스쿨, 모두의 연구소
4
이 발표는?
•범위
- Policy Gradient에서 A2C에 이르는 이론적 배경
- 실습을 통한 A2C 구현의 이해 (Pytorch)
•목표
- Pendulum 환경에서의 에이전트 제어





5
목차
1. Policy Gradient
2. Reinforce
3. Actor Critic (AC)
4. Hands-on Practice: A2C
Policy Gradient
Q. 강화학습?
문제를 풀기 위한
최적의 정책을 찾아내는 것
π⋆
(a|s) ≈ π(a|s; θ)
DQN에서의 최적 정책
π⋆
(a|s) = arg max
a
Q⋆
(s, a) ≈ arg max
a
Q(s, a|θ)
하지만…
•Partially observable state만을 획득할 수 있다면?
- e.g. 포커게임


•Infinite action space를 다뤄야한다면?
- e.g. 로봇팔 제어


12
상상해봅시다
https://kr.pokernews.com/news/2013/09/hand-motions-to-catch-bluffs-kr-5709.htm


https://etinow.me/143
13
Short corridor with switched actions
Sutton, R. and Barto, A. (2018). Reinforcement Learning: An Introduction. 2nd ed. MIT Press
• 입력되는 모든 state가 동일
• 가운데 state에서는 action의 효과가 뒤바뀜
ϵ = 0.1
• 최적의 policy는 좌 : 우를 0.41 : 0.59 비율로 선택하는 것
Q. 강화학습?
A. 문제를 풀기 위한
최적의 정책을 찾아내는 것
적합한 목적함수의 정의!
16
목적함수 정의
CS294-112 at UC Berkeley (2018). Lecture5: Policy Gradient Introduction
17
목적함수 정의
CS294-112 at UC Berkeley (2018). Lecture5: Policy Gradient Introduction
N samples
18
Policy Gradient
CS294-112 at UC Berkeley (2018). Lecture5: Policy Gradient Introduction
19
Policy Gradient
•Finite Horizon Case
∇θJ(θ) = Eτ∼πθ(τ)[(
T
∑
t=1
∇θlog π(at |st))(
T
∑
t=1
r(st, at))]
≈
1
N
N
∑
i
(
T
∑
t=1
∇θlog π(at |st))(
T
∑
t=1
r(st, at))
N samples
20
Policy Gradient
•Markov Property
∇θJ(θ) ≈
1
N
N
∑
i
(
T
∑
t=1
∇θlog π(at |st))(
T
∑
t=1
r(st, at))
t < t’ 일때, t’에서의 policy가 t 시점에 영향을 끼치지 않는다고 가정
=
1
N
N
∑
i
(
T
∑
t=1
∇θlog π(at |st))(
T
∑
t=t′

r(st′

, at′

))
=
1
N
N
∑
i
T
∑
t=1
∇θlog π(at |st) ⋅ Gt
21
Policy Gradient
•Policy Gradient
∇θJ(θ) ≈
1
N
N
∑
i
T
∑
t=1
∇θlog π(at |st) ⋅ Gt
t < t’ 일때, t’에서의 policy가 t 시점에 영향을 끼치지 않는다고 가정
θ ← θ + α∇θJ(θ)
Gradient Ascent!
직관적 해석:
Return이 곱해진 Maximum log likelihood의 형태
- Action probability가 높을수록 업데이트에 덜 반영

- Return이 높을수록 업데이트에 더 반영
22
Policy Gradient
•Reinforce
Sutton, R. and Barto, A. (2018). Reinforcement Learning: An Introduction. 2nd ed. MIT Press
23
Policy Gradient
•Reinforce: High-variance issue
CS294-112 at UC Berkeley (2018). Lecture5: Policy Gradient Introduction
Sample trajectory에 대한 과적합으로 학습을 원활하게 하기 어려움
Variance-Bias trade-o
f
가 필요
24
Policy Gradient
•Reinforce: High-variance issue
CS294-112 at UC Berkeley (2018). Lecture5: Policy Gradient Introduction
Sample trajectory에 대한 과적합으로 학습을 원활하게 하기 어려움
Variance-Bias trade-o
f
가 필요
적절한 Baseline 함수를 이용!
25
Policy Gradient
•Reinforce with Baseline
J(θ) ≈
1
N
N
∑
i
T
∑
t=1
log π(at |st; θ) ⋅ Gt
→
1
N
N
∑
i
T
∑
t=1
log π(at |st; θ) ⋅ (Gt − v(s; w))
theta에 대한 미분과 함께 소거가능
26
Policy Gradient
•Reinforce with Baseline
Sutton, R. and Barto, A. (2018). Reinforcement Learning: An Introduction. 2nd ed. MIT Press
27
Policy Gradient
•Short corridor with switched actions
Sutton, R. and Barto, A. (2018). Reinforcement Learning: An Introduction. 2nd ed. MIT Press
28
Policy Gradient
•Reinforce with Baseline
Sutton, R. and Barto, A. (2018). Reinforcement Learning: An Introduction. 2nd ed. MIT Press
Return 대신 Q(s,a)를 사용한다면?
29
Policy Gradient
•Variance reduction by expectation
Q(S, A)를 G에 대한 기댓값으로 볼 수 있음
30
Policy Gradient
•Actor-Critic
Sutton, R. and Barto, A. (2018). Reinforcement Learning: An Introduction. 2nd ed. MIT Press
Note: Value function을 bootstrapping!
소화하는 시간
실습시간
https://github.com/MrSyee/pg-is-all-you-need
소화하는 시간
감사합니다!

More Related Content

Featured

How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
ThinkNow
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
 

Featured (20)

Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 

Policy Gradient 부터 Actor-Critic 까지