U.S. Army Research, Development and Engineering Command




 A perspective on the role
 of AI and interactive
 digital med...
Disclaimer

• I will be talking about commercial products in addition
  to government-developed technology

• What I say s...
Agenda

•   A little about my organization and my background…
•   Motivation to exploit AI and IDM for military training
•...
My Organization


                         SFC Paul Ray Smith
               Simulation & Training Technology Center

    ...
SFC Paul Ray Smith Simulation & Training Technology Center
A little about me…

• you can find me on

• 25+ years in military training & simulation
  research, development and acquis...
Motivation to exploit AI and Interactive
                                  Media for Military Training




SFC Paul Ray Sm...
Motivation to exploit AI and
                                 Interactive Media for Military Training


•   U.S. Army is d...
Learning & Entertainment


Learning Objectives1
• Gain attention
• Inform                                                 ...
Training

                                         the process of bringing a person,
                                     ...
Exploit AI and Interactive Digital
                                 Media to improve Soldier…


• learning: acquisition of...
Available and emerging
                                          technologies

• intelligent technologies for training
   ...
Intelligent Technologies for Training


PURPOSE: Design, develop, apply and
evaluate intelligent agent technologies
(tools...
Intelligent Technologies for Training



use real behaviors and
     physiological                                        ...
Adaptive Tutoring Research


PURPOSE: Enable computer-based intelligent
tutors to adapt to trainee “readiness-to-learn”
by...
Individual Tutoring Systems




Sottilare, R. (2010). Toward the Development of an Intelligent Tutoring System for Distrib...
Team Tutoring Systems




Sottilare, R. (2010). Toward the Development of an Intelligent Tutoring System for Distributed T...
Adaptive Tutoring Systems


• Few military training simulations today implement
  intelligent tutoring…

• In most of thos...
Virtual Humans




SFC Paul Ray Smith Simulation & Training Technology Center
Virtual Humans

• What parts of real people are represented in virtual
  humans?

   –   physical appearance, behaviors an...
Appraisal Theory

                       courtesy of Stacy Marsella, Institute for Creative Technologies




             ...
Virtual Humans


• Many current and potential applications:

   –    training partners, adversaries or neutral characters
...
Virtual Humans




                                                 courtesy of Jon Gratch, Institute for Creative Technol...
Virtual Patients/Virtual Sick Call



 • Create medically accurate virtual patients
 with advanced interaction capabilitie...
Serious Games




SFC Paul Ray Smith Simulation & Training Technology Center
Tactical Combat Casualty Care




SFC Paul Ray Smith Simulation & Training Technology Center
Tactical Questioning System

PURPOSE:
 Develop a capability for sustainment
 training that included interrogations of
 det...
Multi-national Training Experiment
                                 using Virtual Battlespace 2




• uses commercial game...
UrbanSim

PC-based game that allows Battalion and
Brigade Commanders to practice situational
understanding and operations ...
Bi-lateral Negotiation
                                      Simulation (BiLAT)
PC, game engine-based, cognitive training ...
Bi-lateral Negotiation
                                   Simulation (BiLAT)




                        Social and Cultur...
Virtual Worlds


• Next Generation VW Platforms must be:
   – scalable – work across operating systems and devices
   – ea...
OnLine Interactive
                                   Virtual Environment (OLIVE)




SFC Paul Ray Smith Simulation & Trai...
Multinational Training Experiment
                                     using OLIVE




SFC Paul Ray Smith Simulation & Tra...
Enhanced Dynamic Geo-social
                                            Environment (EDGE)
Blends traits from MMO, serious...
Federal Virtual Worlds Challenge

• What is the Federal Virtual Worlds Challenge?

• It is an open, global Challenge to th...
Mobile Learning




SFC Paul Ray Smith Simulation & Training Technology Center
Combat Medic Card Games




SFC Paul Ray Smith Simulation & Training Technology Center
SFC Paul Ray Smith Simulation & Training Technology Center
Combat Hunter Mobile




SFC Paul Ray Smith Simulation & Training Technology Center
Challenges and Next Steps


•   Economic challenges
•   AI for training challenges
•   IDM for training challenges
•   Int...
Economic Challenges

• The U.S. Army is a large population distributed
  worldwide and covering a variety of occupational
...
AI for Training Challenges

• Adaptable tutoring systems

   – improve perception/prediction of
     trainee’s state

   –...
IDM for Training Challenges

• Interactive mixed reality
  training environments
   – everyone wants the holodeck,
     bu...
Interaction Design
                               Challenges for AI and IDM

• Trainee situational awareness
  comparable ...
Grand Challenges for
                                Educational Technology

•    Personalize Education
•    Assess Studen...
Next Steps for
                                Educational Technology

• User Modeling
• Mobile Training
      – Sensing t...
Questions




SFC Paul Ray Smith Simulation & Training Technology Center
Upcoming SlideShare
Loading in...5
×

2010 10 Sottilare AIIDE Keynote Presentation

629

Published on

Bob\'s presentation AI and Interactive Digital Entertainment (AIIDE) Conference 2010

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
629
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

2010 10 Sottilare AIIDE Keynote Presentation

  1. 1. U.S. Army Research, Development and Engineering Command A perspective on the role of AI and interactive digital media in military training Presentation to the AI and Interactive Digital Entertainment Conference (AIIDE) October 2010 Stanford University Robert A. Sottilare, Ph.D. Chief Technology Officer SFC Paul Ray Smith Simulation & Training Technology Center robert.sottilare@us.army.mil
  2. 2. Disclaimer • I will be talking about commercial products in addition to government-developed technology • What I say should not be construed as an endorsement of any commercial products • Much of what will be covered in this talk has application outside of a military context • Challenge is the word of the day… SFC Paul Ray Smith Simulation & Training Technology Center
  3. 3. Agenda • A little about my organization and my background… • Motivation to exploit AI and IDM for military training • AI and IDM in serious games and military training • Opportunities to exploit AI and IDM • Challenges and Next Steps SFC Paul Ray Smith Simulation & Training Technology Center
  4. 4. My Organization SFC Paul Ray Smith Simulation & Training Technology Center part of the Army Research Laboratory SFC Paul Ray Smith Simulation & Training Technology Center
  5. 5. SFC Paul Ray Smith Simulation & Training Technology Center
  6. 6. A little about me… • you can find me on • 25+ years in military training & simulation research, development and acquisition • My research interests are in adaptive tutoring • my research is conducted in the Learning in Intelligent Tutoring Environments (LITE) Lab • and yes… I like the Simpsons LITE Lab SFC Paul Ray Smith Simulation & Training Technology Center
  7. 7. Motivation to exploit AI and Interactive Media for Military Training SFC Paul Ray Smith Simulation & Training Technology Center
  8. 8. Motivation to exploit AI and Interactive Media for Military Training • U.S. Army is deployed worldwide on a consistent basis so training technology needs to be: – accessible wherever the Soldier is located – interactive regardless of infrastructure available – intuitive so it can be used in the absence of instructors • The time allotted for training is limited so training needs to be: – efficient as well as effective – engaging, challenging and relevant to the Soldier’s mission – adaptive to the trainee’s needs and capabilities Today, Soldiers do more than their traditional missions… SFC Paul Ray Smith Simulation & Training Technology Center
  9. 9. Learning & Entertainment Learning Objectives1 • Gain attention • Inform Stimulate immersion • Stimulate recall Gain/maintain attention • Present stimulus material Motivate • Provide learner LEARNING FUN Guide guidance • Elicit performance Stimulate interaction • Provide feedback Inform • Assess performance • Enhance retention transfer [1] Gagne, R. M. (1965). The conditions of learning. New York: Holt, Rinehart and Winston. SFC Paul Ray Smith Simulation & Training Technology Center
  10. 10. Training the process of bringing a person, team or organization to an agreed standard of proficiency by practice and instruction • relevant task and conditions • standards • measures of performance • constructive and timely feedback SFC Paul Ray Smith Simulation & Training Technology Center
  11. 11. Exploit AI and Interactive Digital Media to improve Soldier… • learning: acquisition of knowledge and/or skill • performance: accomplishment of a given task measured against some standard • retention: ability to perform a learned behavior (task) over time without the reinforcement of practice • competence: demonstrated level of ability through performance over time; potential for future performance SFC Paul Ray Smith Simulation & Training Technology Center
  12. 12. Available and emerging technologies • intelligent technologies for training – adaptive tutoring systems – virtual humans • serious games • virtual worlds • mobile learning SFC Paul Ray Smith Simulation & Training Technology Center
  13. 13. Intelligent Technologies for Training PURPOSE: Design, develop, apply and evaluate intelligent agent technologies (tools and methods) to enhance training, leader development and education; and to reduce associated support costs OBJECTIVE: Use intelligent agents to supplement one-to-one and one-to-many TLE experiences where human support is Adaptive Tutors limited, impractical or completely unavailable TECHNICAL CHALLENGES: Application of intelligent agents in ill-defined domains; understanding natural language in multi- sided conversations; interrupted conversations team tutoring; rapid authoring; and virtual human cognition Virtual Humans SFC Paul Ray Smith Simulation & Training Technology Center
  14. 14. Intelligent Technologies for Training use real behaviors and physiological use simulated measures to model and cognitive states to predict trainee drive virtual human cognitive states behaviors Adaptive Tutors Virtual Humans SFC Paul Ray Smith Simulation & Training Technology Center
  15. 15. Adaptive Tutoring Research PURPOSE: Enable computer-based intelligent tutors to adapt to trainee “readiness-to-learn” by evaluating the influence of performance variables in the trainee model OBJECTIVE: Experimentally evaluate trainee state variables (self-reported, observed, physiological and behavioral) to accurately/ passively assess the trainee’s cognitive state & adapt content/interaction to meet trainee needs TECHNICAL CHALLENGES: Determining optimal sensor suite to accurately/passively identify cognitive state; development of machine learning models to adapt instruction sensors  predicted trainee state  instructional strategy selection SFC Paul Ray Smith Simulation & Training Technology Center
  16. 16. Individual Tutoring Systems Sottilare, R. (2010). Toward the Development of an Intelligent Tutoring System for Distributed Team Training through Passive Sensing. In Proceedings of the 10th Intelligent Tutoring Systems Conference, Pittsburgh, June 2010. SFC Paul Ray Smith Simulation & Training Technology Center
  17. 17. Team Tutoring Systems Sottilare, R. (2010). Toward the Development of an Intelligent Tutoring System for Distributed Team Training through Passive Sensing. In Proceedings of the 10th Intelligent Tutoring Systems Conference, Pittsburgh, June 2010. SFC Paul Ray Smith Simulation & Training Technology Center
  18. 18. Adaptive Tutoring Systems • Few military training simulations today implement intelligent tutoring… • In most of those cases, the trainee model is mainly focused on performance goals and know little or nothing about: – trainee cognition (including affect) – trainee past performance (competence level) • Many of today’s tutors focus in well-defined domains (e.g. physics, math) and the world is messy… • The difficult work is ahead of us… SFC Paul Ray Smith Simulation & Training Technology Center
  19. 19. Virtual Humans SFC Paul Ray Smith Simulation & Training Technology Center
  20. 20. Virtual Humans • What parts of real people are represented in virtual humans? – physical appearance, behaviors and interaction – limited cognition (including affect) – culture, values? – goals, beliefs – social interaction – trust – persistence… Arnold, Lazarus, Frijda, Scherer, Ortony et al. SFC Paul Ray Smith Simulation & Training Technology Center
  21. 21. Appraisal Theory courtesy of Stacy Marsella, Institute for Creative Technologies Arnold, Lazarus, Frijda, Scherer, Ortony et al. SFC Paul Ray Smith Simulation & Training Technology Center
  22. 22. Virtual Humans • Many current and potential applications: – training partners, adversaries or neutral characters – tutoring, mentoring, coaching – marketing – social support – entertainment SFC Paul Ray Smith Simulation & Training Technology Center
  23. 23. Virtual Humans courtesy of Jon Gratch, Institute for Creative Technologies SFC Paul Ray Smith Simulation & Training Technology Center
  24. 24. Virtual Patients/Virtual Sick Call • Create medically accurate virtual patients with advanced interaction capabilities • Train Combat Medics on the proper procedures for sick call TECHNICAL CHALLENGES: • Identifying/creating appropriate and meaningful medical dialog/content • Developing adaptive, realistic human behaviors SFC Paul Ray Smith Simulation & Training Technology Center
  25. 25. Serious Games SFC Paul Ray Smith Simulation & Training Technology Center
  26. 26. Tactical Combat Casualty Care SFC Paul Ray Smith Simulation & Training Technology Center
  27. 27. Tactical Questioning System PURPOSE: Develop a capability for sustainment training that included interrogations of detainees, civilians and persons of interest OBJECTIVE: • Develop 3D Interactive environment • Soldier interacts with virtual humans to sustain their interrogation skills in realistic manner • Utilize translator and foreign languages TECHNICAL CHALLENGES: • Speech and natural language processing • Question and Answer type of conversations • Authoring capability to create new characters SFC Paul Ray Smith Simulation & Training Technology Center
  28. 28. Multi-national Training Experiment using Virtual Battlespace 2 • uses commercial game • primarily kinetic operations • distributed shared environment SFC Paul Ray Smith Simulation & Training Technology Center
  29. 29. UrbanSim PC-based game that allows Battalion and Brigade Commanders to practice situational understanding and operations in complex, contemporary environments. Key Learning Objectives: – Develop and maintain situational understanding – Develop Commander’s intent – Develop Commander’s critical information requirements – Direct operations – Anticipate 2nd and 3rd order effects – Command outside the chain of command SFC Paul Ray Smith Simulation & Training Technology Center
  30. 30. Bi-lateral Negotiation Simulation (BiLAT) PC, game engine-based, cognitive training tool used for developing skills in how to plan for and conduct bi-lateral meetings and/or negotiations in different cultural settings Key Learning Objectives: Negotiation Strategy: Planning and executing tactics that help achieve mission objectives Cultural Awareness: Applying culturally- appropriate interpersonal skills Trust-Building Strategy: Developing/applying tactics to gain negotiating partners’ trust Challenge – combine pedagogy, game play design, interactive research technologies, and software development to address key objectives SFC Paul Ray Smith Simulation & Training Technology Center
  31. 31. Bi-lateral Negotiation Simulation (BiLAT) Social and Cultural Modeling SFC Paul Ray Smith Simulation & Training Technology Center
  32. 32. Virtual Worlds • Next Generation VW Platforms must be: – scalable – work across operating systems and devices – easily deployed – secure – open – interoperable – future proof - scale dynamically – economically sound – good business model David Smith, Chief Innovation Officer, Lockheed Martin Global Training and Logistics A Virtual Worlds Architecture Framework: A Shorter Path to Mastery for All DOD Personnel SFC Paul Ray Smith Simulation & Training Technology Center
  33. 33. OnLine Interactive Virtual Environment (OLIVE) SFC Paul Ray Smith Simulation & Training Technology Center
  34. 34. Multinational Training Experiment using OLIVE SFC Paul Ray Smith Simulation & Training Technology Center
  35. 35. Enhanced Dynamic Geo-social Environment (EDGE) Blends traits from MMO, serious games and virtual world technology to replicate the operational environment as accurately as possible. SFC Paul Ray Smith Simulation & Training Technology Center 3
  36. 36. Federal Virtual Worlds Challenge • What is the Federal Virtual Worlds Challenge? • It is an open, global Challenge to the world for the best of the best implementations within virtual environments. • The event is conducted to explore innovative and interactive solutions in virtual environments. The criteria are intentionally unbounded to encourage creative results. • This year the focus is on artificial intelligence! • http://www.fvwc.army.mil/index.php SFC Paul Ray Smith Simulation & Training Technology Center
  37. 37. Mobile Learning SFC Paul Ray Smith Simulation & Training Technology Center
  38. 38. Combat Medic Card Games SFC Paul Ray Smith Simulation & Training Technology Center
  39. 39. SFC Paul Ray Smith Simulation & Training Technology Center
  40. 40. Combat Hunter Mobile SFC Paul Ray Smith Simulation & Training Technology Center
  41. 41. Challenges and Next Steps • Economic challenges • AI for training challenges • IDM for training challenges • Interaction design challenges • Grand challenges for educational technology • Next steps for educational technology SFC Paul Ray Smith Simulation & Training Technology Center
  42. 42. Economic Challenges • The U.S. Army is a large population distributed worldwide and covering a variety of occupational specialties – Maintaining the proficiency of this population is challenging and expensive – New missions = new training content – New personnel and equipment = continuous training • AI technology (tools and methods) are needed to: – reduce training costs – while maintaining training effectiveness – and improving availability and adaptability SFC Paul Ray Smith Simulation & Training Technology Center
  43. 43. AI for Training Challenges • Adaptable tutoring systems – improve perception/prediction of trainee’s state – improve assessment of learning, performance, competence and retention – improve selection of appropriate instructional strategies based on the trainee’s state (e.g. competency level) • content, tempo, challenge, flow and feedback – automate authoring of content and expert models SFC Paul Ray Smith Simulation & Training Technology Center
  44. 44. IDM for Training Challenges • Interactive mixed reality training environments – everyone wants the holodeck, but its not economically or technically feasible now – explore alternative evolutionary steps to get holodeck-like capabilities now – natural locomotion – no adverse effects (e.g. simulator sickness) – real-time, distributed effects SFC Paul Ray Smith Simulation & Training Technology Center
  45. 45. Interaction Design Challenges for AI and IDM • Trainee situational awareness comparable to “live” environments • Passive machine perception of trainee and “live” environment • Natural Language Understanding & Generation • Natural locomotion in virtual environments SFC Paul Ray Smith Simulation & Training Technology Center
  46. 46. Grand Challenges for Educational Technology • Personalize Education • Assess Student Learning • Support Social Learning • Diminish Boundaries • Develop Alternative Teaching Methods • Enhance the Role of Stakeholders • Address Policy Changes Woolf, B. P. (2010). A Roadmap for Education Technology. National Science Foundation # 0637190 SFC Paul Ray Smith Simulation & Training Technology Center
  47. 47. Next Steps for Educational Technology • User Modeling • Mobile Training – Sensing trainee state during individual training – Interactive team training and performance assessment • Networking Tools • Serious Games • Intelligent Environments • Educational Data Mining (authoring tools) – Content – Models (student models, expert models…) • Rich Interfaces Adapted from: Woolf, B. P. (2010). A Roadmap for Education Technology. National Science Foundation # 0637190 SFC Paul Ray Smith Simulation & Training Technology Center
  48. 48. Questions SFC Paul Ray Smith Simulation & Training Technology Center
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×