Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

Intelligent Online Case Based Planning Agent Model For Rts Games Conference Presentation

on

  • 1,802 views

 

Statistics

Views

Total Views
1,802
Views on SlideShare
787
Embed Views
1,015

Actions

Likes
0
Downloads
1
Comments
0

4 Embeds 1,015

http://omarsbrain.wordpress.com 1010
http://webcache.googleusercontent.com 2
http://translate.googleusercontent.com 2
url_unknown 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Welcome Everybody !My Name is Omar Enayet, I will talk to you about our paper which is entitled “” , Its Authors are … , and from the …
  • Here’s our AgendaWe start by giving and introduction, and viewing the problem definition and the objectives.Then we explain our proposed Intelligent OLCBP Agent model and how was OLCBP hybridized with RLThen we explain an experiment and show the ResultsThen we show the Conclusion and declare the future work
  • So let’s start by giving an introduction,Our papers is mainly based on Online Case Based Planning which is an architecture based on CBR, As you can see the “World” describes the domain in which we need to apply it which could be any domain that requires planning, a computer game or other stuff, then comes the Expansion/Execution Module in which the Solution Tree is Manipulaized, the Expansion Module “problem” nodes which need solution and the Execution Module Executes manages the execution of a solution.On the other side, ….Darmok is a system that applies this without Revising to RTS Games. So the goal of this paper is to Revise or evaluate the cases based on real-time interaction using Reinforcement Learning !Our system evaluates each case so that online learning is performed and the system learns from experience the certainty factor of each case in order to retrieve the most suitable case according to it.
  • Sarsa is a reinforcement learning technique , it’s the abbreviation of STATE-ACTION-REWARD-STATE-ACTION in which the agent after executing a certain action in a certain state recieves a certain reward that helps it choose better its action on its next state.Eligibility Traces are used to make each of the past states responsible or (eligible) for the getting the current reward or the current state.Sarsa lambda combines both to ….. Its an on-policy method that approximates the action values for the current policy then improve the policy gradually based on the approximate values for the current policy.
  • Notice that, cases BuildArmy1 and BuildArmy2 share identical game states, though they contain different plans for achieving the same goal “Build Army”. On the other hand, cases Attack1 and Attack2 achieve the same goal but with different plans, and different game states which are the same as the game states achieved after executing BuildArmy1 and BuildArmy2 respectively. Using BuildArmy1 will definitely force the agent to use Attack1 as BuildArmy1 trains the necessary army that will be used in Attack1. Similarly, using BuildArmy2 will definitely force the agent to use Attack2. It’s known in the game of Wargus, that using heavy units - such as ballista and knights- to attack a towers defense is more effective than using light units such as footmen and archers. This means that it is highly preferable to use case BuildArmy2 instead of case BuildArmy1, and use Attack2 which will definitely cause the agent to destroy more of the enemies units and thus approach wining the game. The experiment constitutes tracing the agent’s evaluation for the cases (after achieving goals “Build Army” then “Attack” in order) for 40 successive times.Notice that BuildArmy1 and BuildArmy2 are rewarded similarly however; the rewards of Attack1 and Attack2 vary greatly due to the different results of both.

Intelligent Online Case Based Planning Agent Model For Rts Games  Conference Presentation Intelligent Online Case Based Planning Agent Model For Rts Games Conference Presentation Presentation Transcript

  • Intelligent Online Case Based Planning Agent Model for RTS Games
    Ibrahim Fathy, MostafaAref, Omar Enayet, and Abdelrahman Al-Ogail
    Faculty of Computer and Information Sciences
    Ain-Shams University ; Cairo ; Egypt
  • Introduction.
    Problem definition.
    Objectives.
    Intelligent OLCBP Agent Model.
    OLCBPRL Hybridization .
    Experiments and Results.
    Conclusion & Future Work.
    Agenda
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Online Case Based Planning is an architecture based on CBR.
    It addresses issues of plan acquisition, on-line plan execution, interleaved planning and execution and on-line plan adaptation.
    Darmok is a previous system that applies this -without revision- to RTS Games.
    Introduction: Online Case-Based Planning
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • A Reinforcement Learning Approach.
    Combines temporal difference learning technique “One-Step SARSA” with eligibility traces to learn state-action pair values effectively.
    It’s an on-policy method.
    Introduction: SARSA(λ) Learning
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Considered a challenging domain due to: Severe Time Constraints – Real-Time AI – Many Objects – Imperfect Information – Micro-Actions
    Introduction: Real-Time Strategy Games
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Research in learning and planning in real-time strategy (RTS) games is very interesting.
    The research on online case-based planning in RTS Games does not include the capability of online learning from experience.
    The knowledge certainty remains constant, which leads to inefficient decisions.
    Problem Definition
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Proposing an intelligent agent model based on both online case-based planning (OLCBP) and reinforcement learning (RL) techniques.
    Increasing the certainty of the case base by learning from experience.
    Increasing both efficiency and effectiveness of the plan decision making process.
    Evaluating the model using empirical simulation on Wargus. ( A clone of the well-known strategy game Warcraft 2)
    Objectives
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Agent Architecture – Abstract View
    Environment ( RTS Game “Wargus”)
    Traces
    Actions
    Offline Phase (Case Acquisition)
    Plan Expansion/Execution Module
    Online Case-Based Learner
    Behaviors
    Case Base
    Retrieved Case
    Cases
    Online Phase
    Goals
    Evaluated Case
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Agent Architecture – Detailed View
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Case Representation
    Case
    Goal
    Strategy
    Situation
    State
    Shallow Features
    Deep Features
    Behavior
    Pre-Conditions
    Alive-Conditions
    Success-Conditions
    Snippet
    Learning Parameters
    Certainty Factor
    Eligibility
    Prior Confidence
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Case Representation - Example
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Case Retrieval
    Since the evaluation of the case base is changed, the case retrieval algorithm must change also.
    The case with the best predicted performance will be retrieved to be executed.
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • OLCBP/RL Hybridization Algorithm
    RL Algorithm used: SARSA(λ)
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • OLCBP/RL Hybridization: The Mapping.
  • Experiment: Case Study
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Results of Attack1 & Attack2
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Results of BuildArmy1 & BuildArmy2
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Agent has learnt that building a smaller heavy army in that specific situation (the existence of a towers defense) is more preferable than building a larger light army. Similarly, the agent can evaluate the entire case base and learn the right choices.
    Results (Cont’d)
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Online case-based planning was hybridized with reinforcement learning in order to introduce an intelligent agent capable of planning and learning online using temporal difference with eligibility traces: Sarsa (λ) algorithm.
    The empirical evaluation has shown that the proposed model –unlike Darmok System - increases the certainty of the case base by learning from experience, and hence the process of decision making for selecting more efficient, effective and successful plans.
    The Paper –Conclusion
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Implementing a prototype based on the proposed model.
    Developing a strategy/case base visualization tool capable of visualizing agent’s preferred playing strategy according to its learning history. This will help in tracking the learning curve of the agent.
    Finally, designing and developing a multi-agent system where agents are able to share their experiences together.
    The Paper –Future Work
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
  • Thank You !
    Questions ?
    Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10