A MODULAR REINFORCEMENT
LEARNING FRAMEWORK
FOR INTERACTIVE NARRATIVE PLANNING
ARTICLE ANALYSIS
Nuno Cancelo
Interactive Ap...
Information
2
The analysis of this thesis was conducted
in the context of the course "Interactive
Applications" from the s...
Information
3
Original Papers
A Modular Reinforcement Learning Framework
for Interactive Narrative Planning
Authors
Jonath...
Summary
 What is
 What for
 How is done
 How They did it
 Policy Induction
 Policy Arbitration
 “Demonstration”
 C...
What is
 Modular Reinforcement Learning Framework?
 Data driven Framework
 Dynamically tailor Events
 Use Modular Rein...
What for
 Generate Interactive Narratives
 Users have a active participation
 Engage user story experience
 Dynamic ta...
How is done
 Decompose the Narrative
 Big story to a sequence of small stories
 Multiple concurrent sub-problems
 Adap...
How They did it
 Based on modular reinforcement learning
 An Agent learn a policy for selecting actions
 The Agent util...
How They did it
 Decomposing Interactive Narratives
 Adaptable event sequence
 A series of multiples related story even...
How They did it
 Four representational considerations
 the overall task must be decomposed
 which involves identifying ...
Policy Induction
 Off-line Reinforcement learning
 Class of procedures separate
 Data collection
 perform narrative ad...
Policy Arbitration
 Multiple decision points are triggered
simultaneously
 receive multiple simultaneous action
recommen...
Where was tested
13
Crystal Island – Outbreak
http://www.intellimedia.ncsu.edu/crystal-island-outbreak/
Crystal Island
14
 src: http://vimeo.com/54292973
Empirical Findings
 75 eighth-grade
 14 were removed
 incomplete or inconsistent data.
 33 students in the Induced Pla...
Empirical Findings
 Induced Planner condition result 61%
(~20)
 Baseline Planner condition result 46%
(~13)
 Students f...
Conclusion
 Simple guidelines
 Very Relevant to this Course
 Easy integration with project (in theory)
 Simple Data st...
Conclusion
 Integration with other authors
 Ex. Rowe, 2013; Sutton and Barto 1998; etc
 Off-Line Reinforcement could be...
Questions/Discussions
19
Contact:
nuno.cancelo@gmail.com
Resources
Crystal Island
 http://ci-lostinvestigation.appspot.com/
 http://www.intellimedia.ncsu.edu/
 http://vimeo.com...
Resources
Markov Decision Process
 http://www.cs.rice.edu/~vardi/dag01/givan1.pdf
 http://courses.cs.washington.edu/cour...
Upcoming SlideShare
Loading in...5
×

Analysis of a modular reinforcement learning framework

184

Published on

Analysis of the paper:
A Modular Reinforcement Learning Framework for Interactive Narrative Planning
from the authors:
Jonathan P. Rowe and James C. Lester

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
184
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Analysis of a modular reinforcement learning framework

  1. 1. A MODULAR REINFORCEMENT LEARNING FRAMEWORK FOR INTERACTIVE NARRATIVE PLANNING ARTICLE ANALYSIS Nuno Cancelo Interactive Applications
  2. 2. Information 2 The analysis of this thesis was conducted in the context of the course "Interactive Applications" from the summer semester of 2013/2014 in the ISCTE-IUL License: Attribution 3.0 Unported (CC BY 3.0) http://creativecommons.org/licenses/by/3.0/
  3. 3. Information 3 Original Papers A Modular Reinforcement Learning Framework for Interactive Narrative Planning Authors Jonathan P. Rowe and James C. Lester Department of Computer Science, North Carolina State University, Raleigh, NC 27695, USA {jprowe, lester}@ncsu.edu src: http://www.aaai.org/ocs/index.php/AIIDE/AIIDE13/paper/view/7472
  4. 4. Summary  What is  What for  How is done  How They did it  Policy Induction  Policy Arbitration  “Demonstration”  Conclusions 4
  5. 5. What is  Modular Reinforcement Learning Framework?  Data driven Framework  Dynamically tailor Events  Use Modular Reinforcing Learning 5
  6. 6. What for  Generate Interactive Narratives  Users have a active participation  Engage user story experience  Dynamic tailored to each user needs  Follow a growing tendency of interaction 6
  7. 7. How is done  Decompose the Narrative  Big story to a sequence of small stories  Multiple concurrent sub-problems  Adaptable Event Sequences (AES)  AES modeled as Markov Decision Process (MDP)  Rewards based on User Experience (EX)  Policies definition 7
  8. 8. How They did it  Based on modular reinforcement learning  An Agent learn a policy for selecting actions  The Agent utilizes a context Reward to learn  Maps states to actions  Maximizes accumulated reward  Modeled as Markov Decision Process 8
  9. 9. How They did it  Decomposing Interactive Narratives  Adaptable event sequence  A series of multiples related story events  Narrative adaptation  A concrete event sequence Similar to Riedl and Bulitko “Interactive Narrative: An Intelligent Systems Approach” 9
  10. 10. How They did it  Four representational considerations  the overall task must be decomposed  which involves identifying the set of AESs and goals  Identification of state representation for each sub-problem  identifying categories of state features  Formalization of event sequences  These sequences are action sets for each MDP  Each MDP must be precisely characterized  These criteria shape the state transition model and reward models 10
  11. 11. Policy Induction  Off-line Reinforcement learning  Class of procedures separate  Data collection  perform narrative adaptations  Model operation  Counting state-transition frequencies in the training corpus 11
  12. 12. Policy Arbitration  Multiple decision points are triggered simultaneously  receive multiple simultaneous action recommendations from distinct policies  If they agree  No arbitration is necessary  If the don’t  Arbitration techniques must be employed 12
  13. 13. Where was tested 13 Crystal Island – Outbreak http://www.intellimedia.ncsu.edu/crystal-island-outbreak/
  14. 14. Crystal Island 14  src: http://vimeo.com/54292973
  15. 15. Empirical Findings  75 eighth-grade  14 were removed  incomplete or inconsistent data.  33 students in the Induced Planner  28 students in the Baseline Planner  “Students played until they solved the mystery or the interaction time expired, whichever occurred first” 15
  16. 16. Empirical Findings  Induced Planner condition result 61% (~20)  Baseline Planner condition result 46% (~13)  Students finished faster 16
  17. 17. Conclusion  Simple guidelines  Very Relevant to this Course  Easy integration with project (in theory)  Simple Data structures implementations 17
  18. 18. Conclusion  Integration with other authors  Ex. Rowe, 2013; Sutton and Barto 1998; etc  Off-Line Reinforcement could be a pain  Good Data sources  Policy induction Arbitration is not clear  AES and MDP MUST be well defined 18
  19. 19. Questions/Discussions 19 Contact: nuno.cancelo@gmail.com
  20. 20. Resources Crystal Island  http://ci-lostinvestigation.appspot.com/  http://www.intellimedia.ncsu.edu/  http://vimeo.com/54292973 A Modular Reinforcement Learning Framework for Interactive Narrative Planning  http://www.aaai.org/ocs/index.php/AIIDE/AIIDE13/paper/view/7472 20
  21. 21. Resources Markov Decision Process  http://www.cs.rice.edu/~vardi/dag01/givan1.pdf  http://courses.cs.washington.edu/courses/csep573/11wi/lectures/12-mdp.pdf  http://www.castlelab.princeton.edu/ORF569papers/Powell_ADP_2ndEdition_Chapter%203.pdf  https://www.youtube.com/watch?v=i0o-ui1N35U This presentation http://www.slideshare.net/NunoCancelo/analysis-of-a-modular-reinforcement-learning-framework 21
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×