Concept, description, and product architecture for soft skills development using virtual AI agent and VR

1. King Speech
Project Codename VRAI

2. Goals
Soft skills development
Full immersion
Real scenarios
Assessment with suggestions on how to improve
and ongoing reviews
No humans needed, AI is the guide and teacher

3. Roadmap
Phase 1 Phase 2 Phase 3 Phase 4
• Voice capture
• Voice sentiment
recognition
• MS PowerPoint plugin
• Google Slides plugin
• Voice/audio
assessment
• Site with assessment
information
• Audio-to-text
• Transcript of session
• NLP classification of
subject based on
ontology & relevance
to presentation
• Questions based on
content
• VR simulation for
Android & PC with non
interactive speaking
scenario
• Assessment based on
VR telemetry
• AI assessment of
presentation
• Text-to-audio
• Interactive VR
simulation based on
scenario
• Assessment based on
behavior during
simulation
• Orchestration AI with
dialogue system

4. Use Cases
Phase 1:
• Do presentation in front on PC using voice
• Review assessment of voice tone based on given speech
Phase 2:
• Do presentation and answer given questions in front on PC
• Review assessment of voice tone, content and answers to questions
Phase 3:
• Do presentation using VR headset
• Review assessment of tone, content, answers and movements
Phase 4:
• Do presentation using VR headset and interact with AI actor during performance
• Review assessment of tone, content, answers, movement and behavior

5. 1. Assessment
1.1 Acoustic
Confident Clear
Dynamic Credible
Persuasive Pleasant
1.2 Speech to Text
1.3 NLP
Language Clearness
Contextual Questions
1.4 Body
Heart Rate (Fit Bit)
2. Feedback
2.1 Audio recording processing
2.2 Real time acoustic analysis
2.3 Interactive charts
2.4 Real time feedback (acoustic)
2.5 Pre-defined scenarios execution
2.6 Real time scenario execution
3. Simulation
3.1 Web portal
3.2 Amazon Alexa Skill
3.2 Microsoft Office 365 Power Point
Add-In
3.3 Google Slides Add-In
3.4 Virtual Reality Simulation
(Samsung Gear VR, Google Card,
Oculus Rift)
3.5 Integration: Google Slides + VR
EVALUATION
REFLACTION
REHEARSAL
MVP 1 MVP 2 MVP 3
Phase 1 Details

6. Engagement Strategy
Phase 1
Voice
Phase 2
Content
Phase 3
Movement
Phase 4
Behavior

7. Architecture – Phase 1
PowerPoint
KS Plugin
Audio
capture
Storage
Classification /
Machine
Learning
Patterns /
Sets
Content &
Timing
Results
Server
Web
Application

8. Architecture – Phase 4
Audio
capture
Storage
Classification /
Machine
Learning
Patterns /
Sets
Content &
Timing
Results
Server
Web
Application
VR Client
Orchestratio
n AI
NLP
Processing
Telemetry
3D Engine
Models &
Animations
Action to Audio
& Action to
activity
Scenario
Ontology
Dialogue
System
Action
Processor
Personality &
Non-linear
behavior

9. VR Technology
Unity based
Android client – medium detail for Google
Cardboard & Samsung Gear VR
Desktop/VR client – full immersion for Oculus Rift,
HTC Vue
Compatibility with Steam and other VR markets
Personal and corporate deployment
Highly detailed models with interactive elements

10. Server Technology
Audio Content AI
Audio analysis & assessment
Audio-to-text
Text-to-audio
Person identification
Sentiment detection
Subject detection
Question asking & answering
Dialogue system
Scenario based dialogue
Character modelling
(passive/neutral/aggressive)
Content assessment
Situation assessment
Scenario based behavior
Fuzzy behavior
Personality based behavior
Scenario assessment
Orchestration AI
Machine Learning for acoustic
analysis

11. Content & situation analysis
How relevant to actual content was what presenter said?
How questions were addressed?
How well presentation was structured?
Which format it followed? Is it repeatable?
Were all necessary topics covered?
What was the reaction during unexpected event (objection
from listeners)?
Did presenter followed dialog structure? Usage of specific
conversational tactics?
How to improve?

12. Potential “soft” pivots
Live voice morphing using voice footprint of other person
(B2C) – e.g. MSQRD for voice
Automated meetings logger (who said what & when with
automatic action items identification) (B2C/B2B)
Extraction of personal voice footprint with future
possibility to “search” in other audio feeds
Integration with bio sensors (hear rate, blood pressure)
for full report on how stressful the presentation was
Assessment of already recorded audio streams (silent
post-review)

13. Platform
Scenario modeling:
General presentation
Public speaking
Interview
Negotiations
Assessment of person
Job scenario simulation (onboarding, client sales
– insurance/bank, review, …)
And many others!