The document discusses building an intelligent virtual personal assistant (VPA) using voice and natural language understanding capabilities. It proposes an architecture for a VPA that uses various Amazon Web Services like Amazon Lex, Amazon Polly, Amazon Comprehend and Amazon Neptune. The architecture includes components for natural language understanding, dialog management, knowledge representation and reinforcement learning to improve the VPA's abilities over time. The presentation concludes that integrating these existing tools could enable the development of an effective interactive VPA to help with knowledge workers' tasks.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Build an Intelligent Multi-Modal User Agent
with Voice and NLU
Keith Steward, Ph.D.
ML Specialist Solutions Architect
Amazon Web Services
A I M 3 4 0

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Questions we’ll address
1. Why is an intelligent agent or virtual personal assistant
(VPA) needed?
2. What capabilities should a VPA have to be helpful?
3. What architecture might support building an intelligent
VPA?
4. Is there a demo of current progress?
5. Q&A

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The problem
• Individuals expected to use a growing number of
online tools and information sources to make
timely decisions and actions
• The complexity, time consumed, and “cognitive
load” associated with these activities is
demanding
• To reduce the workload and amplify individual
productivity, there’s a need to find a way to
quickly and easily delegate these activities to a
VPA/intelligent agent

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Capabilities a VPA might have
1. A desktop agent with visual presence, as
frequent reminder help is available
2. High-bandwidth conversational interaction
between human and agent, including voice and
body (facial) language
3. Reasons about best online resources to address
user’s high-level work needs and
research. Retrieves data, organizes and presents
it, and supports interactive manipulation on
behalf of user
4. Manages a user’s schedule and communications
with colleagues
5. Learns user’s preferences over time; personalizes
responses and actions

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agent backbone (bus)
Text to
Speech
Engine
Speech to
Text
Engine
Tasks
NLU
Engine
Text NLP
Engine
Image &
Face
Analysis
Engine
Semantic
Net/
Knowledge
Graph
Engine
Rules-
based
Inferencing
Engine
Learning &
Personal-
ization
Engine
Calendar
Engine
Modality
Manager
Dialog
Manager
Context
Manager
Service
Manager
“Delight”
Manager
Logical architecture
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agent backbone (bus)
Modality
Manager
Dialog
Manager
Context
Manager
Service
Manager
“Delight”
Manager
Implementation architecture
Amazon
Polly
Amazon
Transcribe
Amazon
Lex
Amazon
Comprehend
CalendarCLIPSAmazon
Neptune
Amazon
Rekognition
Amazon
SageMaker
Text to
Speech
Engine
Speech to
Text
Engine
Tasks
NLU
Engine
Text NLP
Engine
Image &
Face
Analysis
Engine
Semantic
Net /
Knowledge
Graph
Engine
Rules-
based
Inferencing
Engine
Learning &
Personaliz
-ation
Engine
Calendar
Engine
Reinforcement
learning
(Q-learning, OpenAI
Gym/Universe)
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Neptune: Semantic net/
Knowledge Graph engine
• Knowledge Graph containing classes,
entities, attributes, relationships
• Helps agent reason by providing common-
sense knowledge about the world
• Candidate knowledge content
• Freebase
• WordNet
• Lexipedia
• Wikidata
• Open Mind Common Sense (OMCS)
• Universal Networking Language (UNL)

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Rules-based inferencing engine
• Use CLIPS rules-based inferencing
engine as “guide rails” or logic for
basic agent behavior
• Use CLIPS inferencing rules and
assertions about data sources to map
from user’s questions to relevant data
sources for question answering
• User-specific semantic net and CLIPS
assertions to accumulate knowledge
of user over time and reason about
the user

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reinforcement learning engine
• Use reinforcement learning (RL) systems (for
example, Q-learning) with ongoing
observation of results and users’ response to
improve behavior over time
• As RL becomes more optimal, let it drive
increasing amounts of behavior over time (for
example, as accuracy reaches threshold, rules
engine can relax control)

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OpenAI Universe: Assisting user with desktop work
• Universe shares the user’s desktop with agent using
Virtual Network Computing (VNC), a standard
• Universe allows agent to be a VNC client and
observe pixels on desktop, plus interact with the
desktop by producing keyboard and mouse actions

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Conclusions
1. There is a growing need to augment capacity of
knowledge workers
2. An interactive user agent/assistant model might provide
help, allow for delegation of tasks
3. Many powerful components for an interactive user agent
now exist
4. Integration of these components is the key to synergy,
high performance, and rapid extension of new
capabilities
5. We’ve discussed an architecture and useful components,
and seen some progress on implementation

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Keith Steward, Ph.D.

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.