More Related Content
What's hot
What's hot (20)
Similar to All about A.I SuperMario (Reinforcement Learning)
Similar to All about A.I SuperMario (Reinforcement Learning) (7)
Recently uploaded
Recently uploaded (20)
All about A.I SuperMario (Reinforcement Learning)
- 1. 인공지능 슈퍼마리오의 거의 모든 것 Reinforcement Learning Wonseok Jung
- 2. 정원석 Wonseok Jung City University of New York “- Baruch College (Data Science Major) ConnexionAI A.I Researcher DeepLearningCollege Reinforcement Learning Researcher 모두의연구소 CTRL (Contest in RL) Leader Project what I’ve done : Reinforcement Learning, Object Detection, Chatbot Github: https://github.com/wonseokjung Facebook: https://www.facebook.com/ws.jung.798 Blog: https://wonseokjung.github.io/
- 3. 목차 1. How Animals Learn 2. How Humans Learn 3. Reinforcement Learning 4. SuperMario with Reinforcement Learning REINFORCEMENT LEARNING
- 4. PREVIEW REINFORCEMENT LEARNING Animal Human SuperMario A A Env R AtRt SSt Rt+1 St+1 Reinforcement Learning Agent Environment
- 6. HABUTUATION - 모든 동물은 학습능력이 있다. - 300여개의 신경세포만을 갖고 있는 예쁜꼬마선충 또한 학습능력이 있다. - Head withdrawal reflex : 위험한 물체가 있을것이라 판단에 따른 반사행동 - 예쁜꼬마선충의 머리를 건드리면 일정 거리를 뒤로 간다. HOW ANIMALS LEARN
- 7. HABUTUATION HOW ANIMALS LEARN First try Second try Third try
- 8. LAW OF EFFECT - Edward Thorndike(1898) - Law of effect : 어떤 행동의 결과가 만족스러우면 다음에도 그 행동을 반복한다. 반대로 만족하지 않으면 그 행동을 하지 않는다. - Reinforcement(강화) : 이전에 일어난 행동을 반복하게 만드는 자극 - Punishment(처벌) : 이전에 일어난 행동을 피하게 만드는 자극 HOW ANIMALS LEARN
- 9. EXAMPLE OF LAW OF EFFECT HOW ANIMALS LEARN
- 10. HOW HUMANS LEARN
- 11. INTERACTION WITH ENVIRONMENT REINFORCEMENT LEARNING Environment Experience LearnInteraction
- 12. HOW HUMANS LEARN? - Reinforcement : 이전에 일어난 행동을 반복하게 만드는 자극 - Punishment : 이전에 일어난 행동을 피하게 만드는 자극 HOW HUMANS LEARN
- 13. HOW HUMANS LEARN Experiment Using Tap ball HOW HUMANS LEARN https://www.youtube.com/watch?v=2sicukP34fk
- 14. HOW HUMANS LEARN -TAP BALL Day 1 Day 2 Day 3 Day 4 최고점수 : 3 맞은횟수 : 2 최고점수 : 23 맞은횟수 : 0 최고점수 : 30 맞은횟수 : 0 최고점수 : 38 맞은횟수 : 1 HOW HUMANS LEARN
- 15. HOW HUMAN LEARN Day 5 최고점수 : 79 맞은횟수 : 0 HOW HUMANS LEARN
- 16. SIMILARITY LEARNING METHOD B/W ANIMALS AND HUMAN HOW HUMANS LEARN Punishment Punishment Punishment
- 18. REINFORCEMENT LEARNING Environment Experience Learn REINFORCEMENT LEARNING Interaction
- 19. 용어 정의 Time step Action Transition Function Reward Set of states Set of actions Start state Discount factor t a P(s′, r ∣ s, a) r A S S0 γ Set of reward Policy Reward State R π r REINFORCEMENT LEARNING s
- 20. TERMINATION Time step Action Transition Function Reward Set of states Set of actions Start state Discount factor t a P(s′, r ∣ s, a) r A S S0 γ Set of reward Policy Reward State R π r REINFORCEMENT LEARNING s
- 21. LEARNING - Reinforcement learning은 Reward(보상)을 최대화 하는 action(행동)을 선택한다. - Learner(배우는자)는 여러 action을 해보며, reward를 가장 높게 받는 action을 찾는다. -선택된 action이 당장의 reward 뿐만 아닌, 다음의 상황 또는 다음 일어나게 될 reward에도 영향을 끼칠수도 있다. Action 당장의 상황 변화 미래의 상황Reward 미래의 Reward REINFORCEMENT LEARNING
- 22. MARKOV DECISION PROCESS Action Agent Environment Reward AtRt State St Rt+1 St+1 REINFORCEMENT LEARNING
- 23. TOTAL REWARD REINFORCEMENT LEARNING Return of Episode Return of Episode with discount factor
- 25. STATE-ACTION VALUE FUNCTION State-Action value REINFORCEMENT LEARNING
- 26. OPTIMAL POLICY Optimal State-Value function REINFORCEMENT LEARNING Optimal State-Action value function
- 27. Agent Exploitation Exploration ? EXPLOITATION AND EXPLORATION REINFORCEMENT LEARNING
- 28. IMPORTANCE OF EXPLORATION 셀이 RussianBlue 2살 Curiosity 풀이 Munchkin 1살 Food REINFORCEMENT LEARNING
- 29. IMPORTANCE OF EXPLORATION REINFORCEMENT LEARNING 풀이 셀이 Zero exploration Exploration
- 30. IMPORTANCE OF EXPLORATION-2 REINFORCEMENT LEARNING 풀이 셀이 Fail
- 31. SUPERMARIO WITH REINFORCEMENT LEARNING
- 32. MARKOV DECISION PROCESS Action Agent Environment Reward AtRt State St Rt+1 St+1 SUPERMARIO WITH R.L Reward: +1 Penalty: -1
- 33. MARKOV DECISION PROCESS Action Agent Environment Reward AtRt State St Rt+1 St+1 SUPERMARIO WITH R.L Reward: +1 Penalty: -1
- 34. SUPERMARIO WITH R.L https://github.com/wonseokjung/gym-super-mario-bros pip install gym-super-mario-bros import gym_super_mario_bros env = gym_super_mario_bros.make(‘SuperMarioBros-v0') env.reset() env.render() INSTALL AND IMPORT ENVIRONMENT
- 35. WORLDS & LEVELS SUPERMARIO WITH R.L World 1 World 3 World 2 World 4 env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')
- 37. REWARD AND PENALTY SUPERMARIO WITH R.L Reward Penalty 깃발에 가까워지면 + 목표에 도착하면 + 목표달성하지 못하면 - 시간이 지날때마다 - 깃발에서 멀어지면 -
- 38. STATE, ACTION SUPERMARIO WITH R.L env.observation_space.shape (240, 256, 3) # [ height, weight, channel ] env.action_space.n 256 SIMPLE_MOVEMENT = [ [‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ] from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv import gym_super_mario_bros env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT)
- 39. EXPLOITATION AND EXPLORATION SUPERMARIO WITH R.L next_state, reward, done, info = env.step(action) def epsilon_greedy(q_value,step): if np.random.rand() < epsilon : return np.random.randint(output) else : action = np.argmax(output) Exploitation Exploration
- 40. REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state) (St, At, Rt+1, St+1) next_state, reward, done, info = env.step(action) eps_min = 0.1 eps_max = 1 eps_decay_steps = 200000
- 41. MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf loss = tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss) (Rt+1 + γt+1maxa′qθ(St+1, a′ ) − qθ(St, At))2 (St, At, Rt+1, St+1)
- 42. MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf loss = tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss) (Rt+1 + γt+1maxa′qθ(St+1, a′ ) − qθ(St, At))2 (St, At, Rt+1, St+1)
- 44. 1000EPISODE, 3000EPISODE, TRAINING SUPERMARIO WITH R.L 1000 episode 3000 episode
- 45. 5000 EPISODE SUPERMARIO WITH R.L 5000 episode
- 46. SUMMARY 1. How Animals Learn 2. How Humans Learn 3. Reinforcement Learning 4. SuperMario with Reinforcement Learning REINFORCEMENT LEARNING
- 47. REFERENCES 1. Habituation The Birth of Intelligence 2. Law of effect : The Birth of Intelligence ,p.171 3. Thorndike, E. L. (1905). The elements of psychology. New York: A. G. Seiler. 4. Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative processes in animals. Psychological Monographs: General and Applied, 2(4), i-109. 5. Supermario environment https://github.com/Kautenja/gym-super-mario-bros 6. http://faculty.coe.uh.edu/smcneil/cuin6373/idhistory/thorndike_extra.html
- 48. Question?