The document discusses reinforcement learning and its application to training an AI agent to play Super Mario Bros. It begins by explaining how animals and humans learn through reinforcement and punishment in their environments based on the rewards and consequences of their actions. It then provides an overview of reinforcement learning, including key concepts like the Markov decision process, exploration versus exploitation, and using a replay memory buffer to train a deep learning model. It concludes by describing how the Super Mario Bros environment can be set up and used with a reinforcement learning agent, including defining the state and action spaces, rewards and penalties, and the process of minimizing the loss to optimize the agent's behavior.

1. 인공지능 슈퍼마리오의 거의 모든 것
Reinforcement Learning
Wonseok Jung

2. 정원석
Wonseok Jung
City University of New York - Baruch College (Data Science Major)
ConnexionAI A.I Researcher
DeepLearningCollege Reinforcement Learning Researcher
모두의연구소 CTRL (Contest in RL) Leader
Reinforcement Learning, Object Detection, Chatbot
Github:
https://github.com/wonseokjung
Facebook:
https://www.facebook.com/ws.jung.798
Blog:
https://wonseokjung.github.io/

3. 목차
1. How Animals Learn
2. How Humans Learn
3. Reinforcement Learning
4. SuperMario with Reinforcement Learning
REINFORCEMENT LEARNING

4. PREVIEW
REINFORCEMENT LEARNING
Animal Human SuperMario
A
A
Env
R AtRt
SSt
Rt+1
St+1
Reinforcement
Learning
Agent
Environment

5. HOW ANIMALS LEARN

6. ALL ANIMALS HAVE THE ABILITY TO LEARN
- 모든 동물은 학습능력이 있다.
- 300여개의 신경세포만을 갖고 있는 예쁜꼬마선충 또한 학습능력이 있다.
- 머리철수반사 head withdraws reflex : 위험한 물체가 있을것이라 판단에 따른 반사행동
- 예쁜꼬마선충의 머리를 건드리면 일정 거리를 뒤로 간다.
HOW ANIMALS LEARN

7. HABITUATION
HOW ANIMALS LEARN
First try
Second try
Third try

8. LAW OF EFFECT
- Edward Thorndike(1898)
- Law of effect : 어떤 행동의 결과가 만족스러우면 다음에도 그 행동을 반복한다.
반대로 만족하지 않으면 그 행동을 하지 않는다.
- Reinforcement(강화) : 이전에 일어난 행동을 반복하게 만드는 자극
- Punishment(처벌) : 이전에 일어난 행동을 피하게 만드는 자극
HOW ANIMALS LEARN

9. EXAMPLE OF THE LAW OF EFFECT
HOW ANIMALS LEARN

10. HOW HUMANS LEARN

11. INTERACTION WITH ENVIRONMENT
Environment Experience
LearnInteraction
HOW HUMANS LEARN

12. HOW HUMANS LEARN?
- Reinforcement : 이전에 일어난 행동을 반복하게 만드는 자극
- Punishment : 이전에 일어난 행동을 피하게 만드는 자극
HOW HUMANS LEARN

13. HOW HUMANS LEARN
Experiment Using Tap ball
HOW HUMANS LEARN
https://www.youtube.com/watch?v=2sicukP34fk

14. HOW HUMANS LEARN -TAP BALL
Day 1 Day 2 Day 3 Day 4
최고점수 : 3
맞은횟수 : 2
최고점수 : 23
맞은횟수 : 0
최고점수 : 30
맞은횟수 : 0
최고점수 : 38
맞은횟수 : 1
HOW HUMANS LEARN

15. TAP BALL DAY 5
Day 5
최고점수 : 79
맞은횟수 : 0
HOW HUMANS LEARN

16. SIMILAR LEARNING METHODS B/W ANIMALS AND HUMANS
HOW HUMANS LEARN
Punishment Punishment Punishment

17. REINFORCEMENT LEARNING

18. REINFORCEMENT LEARNING
Environment Experience
Learn
REINFORCEMENT LEARNING
Interaction

19. LEARNING
- Reinforcement learning은 Reward(보상)을 최대화 하는 action(행동)을 선택한다.
- Learner(배우는자)는 여러 action을 해보며, reward를 가장 높게 받는 action을 찾는다.
-선택된 action이 당장의 reward 뿐만 아닌, 다음의 상황 또는 다음 일어나게 될
reward에도 영향을 끼칠수도 있다.
Action
당장의
상황 변화
미래의 상황Reward 미래의 Reward
REINFORCEMENT LEARNING

20. Agent
Exploitation Exploration
?
EXPLOITATION AND EXPLORATION
REINFORCEMENT LEARNING

21. IMPORTANCE OF EXPLORATION
셀이
RussianBlue
2살
Curiosity
풀이
Munchkin
1살
Food
REINFORCEMENT LEARNING

22. IMPORTANCE OF EXPLORATION-2
REINFORCEMENT LEARNING
풀이 셀이
Zero
exploration
Exploration

23. IMPORTANCE OF EXPLORATION-3
REINFORCEMENT LEARNING
풀이 셀이
Fail

24. MARKOV DECISION PROCESS
Action
Agent
Environment
Reward
AtRt
State
St
Rt+1
St+1
REINFORCEMENT LEARNING

25. MARKOV DECISION PROCESS
Environment
Reward
AtRtSt
Rt+1
St+1
REINFORCEMENT LEARNING
Tap the ball
Positive Reward

26. STATE-VALUE FUNCTION
State-value
REINFORCEMENT LEARNING

27. STATE-ACTION VALUE FUNCTION
State-Action value
REINFORCEMENT LEARNING

28. OPTIMAL POLICY
Optimal State-Value function
REINFORCEMENT LEARNING
Optimal State-Action value function

29. SUPERMARIO WITH REINFORCEMENT LEARNING

30. MARKOV DECISION PROCESS
Action
Agent
Environment
Reward
AtRt
State
St
Rt+1
St+1
SUPERMARIO WITH R.L
Reward: +1
Penalty: -1

31. MARKOV DECISION PROCESS
Action
Agent
Environment
Reward
AtRt
State
St
Rt+1
St+1
SUPERMARIO WITH R.L
Reward: +1
Penalty: -1

32. SUPERMARIO WITH R.L
https://github.com/wonseokjung/gym-super-mario-bros
pip install gym-super-mario-bros
import gym_super_mario_bros
env = gym_super_mario_bros.make(‘SuperMarioBros-v0')
env.reset()
env.render()
INSTALL AND IMPORT ENVIRONMENT

33. WORLDS & LEVELS ( WORLD 1~4)
SUPERMARIO WITH R.L
World 1 World 3
World 2 World 4
env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')

34. WORLDS & LEVELS ( WORLD 5~8)
SUPERMARIO WITH R.L
World 5 World 7
World 6 World 8
env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')

35. ALL WORLDS AND LEVELS
SUPERMARIO WITH R.L
env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')
1
2
3
4
5
6
7
8

36. ALL WORLDS AND LEVELS
SUPERMARIO WITH R.L
env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')
1
2
3
4
5
6
7
8
1 2 3 4 1 2 3 4

37. WORLDS & LEVELS
SUPERMARIO WITH R.L
Version 1
env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')
Version 2 Version 3 Version 4

38. GOAL
SUPERMARIO WITH R.L

39. REWARD AND PENALTY
SUPERMARIO WITH R.L
Reward
Penalty
깃발에 가까워지면 +
목표에 도착하면 +
목표달성하지 못하면 -
시간이 지날때마다 -
깃발에서 멀어지면 -

40. STATE, ACTION
SUPERMARIO WITH R.L
env.observation_space.shape
(240, 256, 3) # [ height, weight, channel ]
env.action_space.n
256
SIMPLE_MOVEMENT = [
[‘nop’],
[‘right’],
[‘right’,’A’],
[‘right’,’B’],
[‘right’,’A’,’B’],
[‘A’],
[‘left’],
]
from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv
import gym_super_mario_bros
env = gym_super_mario_bros.make(‘SuperMarioBros-v0’)
env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT)

41. OBSERVATION SPACE
SUPERMARIO WITH R.L
env.action_space.n
256
SIMPLE_MOVEMENT = [
[‘nop’],
[‘right’],
[‘right’,’A’],
[‘right’,’B’],
[‘right’,’A’,’B’],
[‘A’],
[‘left’],
]
from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv
import gym_super_mario_bros
env = gym_super_mario_bros.make(‘SuperMarioBros-v0’)
env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT)
env.observation_space.shape
(240, 256, 3) # [ height, weight, channel ]

42. ACTION SPACE
SUPERMARIO WITH R.L
env.action_space.n
256
SIMPLE_MOVEMENT = [
[‘nop’],
[‘right’],
[‘right’,’A’],
[‘right’,’B’],
[‘right’,’A’,’B’],
[‘A’],
[‘left’],
]
from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv
import gym_super_mario_bros
env = gym_super_mario_bros.make(‘SuperMarioBros-v0’)
env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT)
env.observation_space.shape
(240, 256, 3) # [ height, weight, channel ]

43. ACTION AFTER WRAPPER
SUPERMARIO WITH R.L
env.action_space.n
256
SIMPLE_MOVEMENT = [
[‘nop’],
[‘right’],
[‘right’,’A’],
[‘right’,’B’],
[‘right’,’A’,’B’],
[‘A’],
[‘left’],
]
import gym_super_mario_bros
env = gym_super_mario_bros.make(‘SuperMarioBros-v0’)
env.observation_space.shape
(240, 256, 3) # [ height, weight, channel ]
env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT)
from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv

44. EXPLOITATION AND EXPLORATION
SUPERMARIO WITH R.L
next_state, reward, done, info = env.step(action)
else :
action = np.argmax(output)
Exploitation Exploration
def epsilon_greedy(q_value,step):
if np.random.rand() < epsilon :
action=np.random.randint(output)
?

45. EXPLORATION
SUPERMARIO WITH R.L
next_state, reward, done, info = env.step(action)
else :
action = np.argmax(output)
Exploitation Explorationif np.random.rand() < epsilon :
action=np.random.randint(output)
def epsilon_greedy(q_value,step):
?

46. EXPLOITATION
SUPERMARIO WITH R.L
next_state, reward, done, info = env.step(action)
else :
action = np.argmax(output)
def epsilon_greedy(q_value,step):
if np.random.rand() < epsilon :
action=np.random.randint(output)
Exploitation Exploration
?

47. ENV.STEP( )
SUPERMARIO WITH R.L
next_state, reward, done, info = env.step(action)
else :
action = np.argmax(output)
def epsilon_greedy(q_value,step):
if np.random.rand() < epsilon :
action=np.random.randint(output)

48. EXPLORATION RATE AND REPLAY MEMORY BUFFER
SUPERMARIO WITH R.L
memory = deque([],maxlen=1000000)
memory.append(state,action,reward,next_state)
(St, At, Rt+1, St+1)
next_state, reward, done, info = env.step(action)
eps_max = 1
eps_min = 0.1
eps_decay_steps = 200000

49. REPLAY MEMORY BUFFER
SUPERMARIO WITH R.L
memory = deque([],maxlen=1000000)
memory.append(state,action,reward,next_state)
next_state, reward, done, info = env.step(action)
eps_max = 1
eps_decay_steps = 200000
eps_min = 0.1

50. REPLAY MEMORY BUFFER
SUPERMARIO WITH R.L
memory = deque([],maxlen=1000000)
memory.append(state,action,reward,next_state)
next_state, reward, done, info = env.step(action)
eps_max = 1
eps_decay_steps = 200000
eps_min = 0.1

51. REPLAY MEMORY BUFFER
SUPERMARIO WITH R.L
memory = deque([],maxlen=1000000)
memory.append(state,action,reward,next_state)
next_state, reward, done, info = env.step(action)
eps_max = 1
eps_decay_steps = 200000
eps_min = 0.1

52. REPLAY MEMORY BUFFER
SUPERMARIO WITH R.L
memory = deque([],maxlen=1000000)
memory.append(state,action,reward,next_state)
eps_max = 1
eps_min = 0.1
eps_decay_steps = 200000
next_state, reward, done, info = env.step(action)

53. REPLAY MEMORY BUFFER
SUPERMARIO WITH R.L
eps_max = 1
eps_min = 0.1
eps_decay_steps = 200000
next_state, reward, done, info = env.step(action)
memory = deque([],maxlen=1000000)
memory.append(state,action,reward,next_state)

54. REPLAY MEMORY BUFFER
SUPERMARIO WITH R.L
eps_max = 1
eps_min = 0.1
eps_decay_steps = 200000
next_state, reward, done, info = env.step(action)
memory.append(state,action,reward,next_state)
memory = deque([],maxlen=1000000)

55. MINIMIZE LOSS
SUPERMARIO WITH R.L
import tensorflow as tf
loss = tf.reduce_mean(tf.squre( y - Q_action ) )
Optimizer =tf.train.AdamsOptimizer(learning_rate)
training_op = optimizer.minize(loss)
(Rt+1 + γt+1maxa′qθ(St+1, a′
) − qθ(St, At))2
(St, At, Rt+1, St+1)

56. MINIMIZE LOSS
SUPERMARIO WITH R.L
(Rt+1 + γt+1maxa′qθ(St+1, a′
) − qθ(St, At))2
(St, At, Rt+1, St+1)
loss = tf.reduce_mean(tf.squre( y - Q_action ) )
Optimizer =tf.train.AdamsOptimizer(learning_rate)
training_op = optimizer.minize(loss)
import tensorflow as tf

57. MINIMIZE LOSS
SUPERMARIO WITH R.L
import tensorflow as tf
Optimizer =tf.train.AdamsOptimizer(learning_rate)
training_op = optimizer.minize(loss)
(Rt+1 + γt+1maxa′qθ(St+1, a′
) − qθ(St, At))2
(St, At, Rt+1, St+1)
loss = tf.reduce_mean(tf.squre( y - Q_action ) )

58. MINIMIZE LOSS
SUPERMARIO WITH R.L
import tensorflow as tf
training_op = optimizer.minize(loss)
(Rt+1 + γt+1maxa′qθ(St+1, a′
) − qθ(St, At))2
(St, At, Rt+1, St+1)
loss = tf.reduce_mean(tf.squre( y - Q_action ) )
Optimizer =tf.train.AdamsOptimizer(learning_rate)

59. MINIMIZE LOSS
SUPERMARIO WITH R.L
import tensorflow as tf
(Rt+1 + γt+1maxa′qθ(St+1, a′
) − qθ(St, At))2
(St, At, Rt+1, St+1)
loss = tf.reduce_mean(tf.squre( y - Q_action ) )
Optimizer =tf.train.AdamsOptimizer(learning_rate)
training_op = optimizer.minize(loss)

60. APPROXIMATE ACTION-VALUE
SUPERMARIO WITH R.L

61. DOUBLE DQN
SUPERMARIO WITH R.L
input
Action
value EnvQ-Network s’
s
Replay memory
Q(s,a)
a
r
(St, At, Rt+1, St+1)

62. 1000EPISODE, 3000EPISODE, TRAINING
SUPERMARIO WITH R.L
1000 episode 3000 episode

63. 5000 EPISODE
SUPERMARIO WITH R.L
5000 episode
4 Days

64. SUMMARY
1. How Animals Learn
2. How Humans Learn
3. Reinforcement Learning
4. SuperMario with Reinforcement Learning
REINFORCEMENT LEARNING

65. OTHER ENVIRONMENTS
OpenAI Deepmind Lab Starcraft
Supermario Sonic Minecraft
REINFORCEMENT LEARNING

66. OTHER LEARNING METHODS
DQN
DDQN(tuned) Rainbow DQN(tuned) DDPG
REINFORCEMENT LEARNING

67. CURRICULUM LEARNING
Goal
Wall
Agent
Action3Action2
Action1
REINFORCEMENT LEARNING

68. IMITATION LEARNING
Imitation Learning
Teacher Student
REINFORCEMENT LEARNING

69. REINFORCEMENT LEARNING

70. How about making your own A.I SuperMario?

71. Github:
https://github.com/wonseokjung
Facebook:
https://www.facebook.com/ws.jung.798
Blog:
https://wonseokjung.github.io/
감사합니다.
Thank you

72. Question?

73. *참고 용어 기호
Time step
Action
Transition Function
Reward
Set of states
Set of actions
Start state
Discount factor
t
a
P(s′, r ∣ s, a)
r
A
S
S0
γ
Set of reward
Policy
Reward
State
R
π
r
REINFORCEMENT LEARNING
s

74. REFERENCES
1. Habituation The Birth of Intelligence
2. Law of effect : The Birth of Intelligence ,p.171
3. Thorndike, E. L. (1905). The elements of psychology. New York: A. G. Seiler.
4. Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative
processes in animals. Psychological Monographs: General and Applied, 2(4), i-109.
5. SuperMario environment
https://github.com/Kautenja/gym-super-mario-bros
6. http://faculty.coe.uh.edu/smcneil/cuin6373/idhistory/thorndike_extra.html