This document discusses reinforcement learning. It begins with basic definitions and applications of reinforcement learning. It then discusses policy based reinforcement learning methods, including value-based methods which estimate the value function and have an implicit policy, and policy gradient methods which directly estimate the policy. Specific algorithms discussed include Q-learning, Sarsa, and policy gradients. Examples applications provided include AlphaGo, robotics, healthcare, online trading, and scheduling.

1. Reinforcement learning
#AIEnsamble
London Artificial Intelligence Meetup

2. Leonardo De Marchi

3. Reinforcement learning
‣ Basic definitions
‣ Applications
‣ Policy based methods
Theory Demo
‣ OpenAI

4. MACHINE LEARNING
SUPERVISED REINFORCEMENTUNSUPERVISED

5. SUPERVISED - TRAINING
INPUT OUTPUT

6. SUPERVISED - TRAINING
INPUT OUTPUTMODEL

7. SUPERVISED - scoring
INPUT OUTPUTMODEL

8. SUPERVISED
- scoring

9. unSUPERVISED
INPUT input clusteredMODEL

10. unSUPERVISED
INPUT input clusteredMODEL

11. Reinforcement
learning

12. Reinforcement learning
agent
environment
feedback
action

13. applications

14. Alpha go The image part with relationship ID rId2 was not found in the file.

15. Alpha zero

16. Robotic

17. Why it matters
‣ Text summarisation engines
‣ Dialog agents (text, speech)
‣ Learning optimal treatment policies in healthcare
‣ Online stock trading
‣ Scheduling
‣ …

18. Why it matters
‣ To learn how to make decisions to achieve a specific goal

19. games

20. A2C

21. GQN

22. GQN

23. https://www.youtube.com/watch?v=oo0TraGu6QY
https://www.youtube.com/watch?time_continue=72&v=TmPfTpjtdgg
https://www.youtube.com/watch?v=UZHTNBMAfAA
https://www.youtube.com/watch?time_continue=118&v=eHipy_j29Xw
Interesting Applications

24. our problem

25. The Problem
agent
environment
feedback
action

26. ‣Markov Decision Process
Agent Environment
at
‣Rt Rt+1 st+1
‣St

27. Environment

28. Environment
‣ Multiple states
‣ Complex reward function

29. Feedback
‣ Returned by the environment
+10
+1

30. Goal
‣ Maximise the total reward

31. Reinforcement
Learning
Algorithms

32. Value-basedmethods
‣ Use policy and expected return to take action
‣ Estimate the value function
‣ Policy is implicit (eg 𝜀-greedy)
‣ i.e. Sarsa, Q-learning

33. Value-basedmethods
‣ Use policy and expected return to take action
‣ Estimate the value function
‣ Policy is implicit (eg 𝜀-greedy)
‣ i.e. Sarsa, Q-learning

34. Value-basedmethods
‣ Use policy and expected return to take action
‣ Estimate the value function
‣ Policy is implicit (eg 𝜀-greedy)
‣ i.e. Sarsa, Q-learning

35. Value-basedmethods
Q-function
policy
environment
feedback
action

36. Policy gradients
𝝅𝜽(s|a)
environment
feedback
action
𝝅 𝜃(a|s) = probability of action a in state s

37. Policy gradients
𝝅𝜽(s|a)
environment
feedback
action
Actions

38. Policy bASEDMETHODS
E=[max
(
∑ 𝑅 𝑠𝑡 |.
/01 𝝅 𝜃]
Policy
If we change an action we have a big impact
Changing the action distribution will have a smaller impact

39. Policy-Based methods
‣ Estimate the policy
‣ No value function
‣ For simpler problems
‣ Innate exploration by
his stochastic nature
‣ Can be used together
with supervised
learning

40. Policy gradient
‣ Recent success in video game, 3d locomotion, and Go
‣ Problems: sensitive to step size
‣ Slow progress
‣ Noise can mask the signal

41. Demo

42. Takeaway
‣ RL is useful
‣ Policy gradients had a lot of success
‣ OpenAI’s gym is a great tool to test RL algorithms

43. Training
Modern Machine Learning and Deep Learning
2 days course covering real life Deep Learning examples
https://www.eventbrite.co.uk/e/modern-machine-learning-and-deep-learning-
2-day-course-tickets-49603205523?aff=ebdssbdestsearch
Use discount code: IDEAIFORME

44. Thank you!
You can contact me at
www.ideai.io info@ideai.io
Newsletter:
subscribe@ideai.io