ALX307_Integrate Alexa into Your Product

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:Invent
Integrate Alexa into Your Product
Using the Alexa Voice Service
Device SDK N o v e m b e r 2 8 , 2 0 1 7

Meet the team!
Chris Keyser
Principal Solutions Architect, Alexa
Voice Service
Sanjay Devireddy
Software Development Engineer,
Alexa Voice Service
Gagan Luthra
Sr. Product Marketing Manager, Alexa
Voice Service
Fletcher Nelson
Solutions Architect, Alexa
Voice Service
Navneet Malpani
Principal Solutions Architect, Alexa
Voice Service

Amazon Echo: The first Alexa-enabled device
“Amazon’s Echo might be the
most important product in years”
“Amazon Echo proves the viability
of voice-based home computing.”
“Amazon Alexa just conquered CES 2017.
The world is next.”
“Alexa is clearly the star of IFA 2017”

Hi, Alexa… Hello!

Smart home
control
Timers and alarms
Cloud-based
intelligence
Shopping
Music
News and
information
Custom skills
Always getting smarter

Amazon’s Alexa-enabled products

Vision: Alexa anywhere
Home audio
Thermostats
Automobiles
Maker boards
Wearables
Small appliances
Smartphones
Large appliancesHome security
TV/video

Alexa Voice Service
A l e x a - e n a b l e d p r o d u c t s

Directives
Instructions sent from AVS to client
Events
Information sent from client to AVS
Voice hardware
Audio front end
Device software
AVS device SDK
Simplifying voice interaction

“Alexa, play music.”
Wake word
engine
Recognition result
Speech
platform
Skills
Music
Service
ASR
NLU
TTS
“Recognize” event
User’s utterance
Event
Alexa Voice Service
Speech
capture via
microphone
Play back
through
speaker

Wake word
engine
Recognition result
Recognition resultSpeech
platform
Skills
Music
Service
ASR
NLU
TTS
User’s utterance
Event
Intent
Alexa Voice Service
Speech
capture via
microphone
Play back
through
speaker

Wake word
engine
“Play” directive
Intent
Recognition result
Recognition resultSpeech
platform
Skills
Music
Service
ASR
NLU
TTS
User’s utterance
Event
Intent
Alexa Voice Service
Speech
capture via
microphone
Play back
through
speaker

Wake word
engine
“Play” directive
Intent
Recognition result
Recognition result
Text/SSML
Speech
platform
Skills
Music
Service
ASR
NLU
TTS
Streaming URL
User’s utterance
Event
Alexa voice response
Directive
Intent
Alexa Voice Service
Speech
capture via
microphone
Play back
through
speaker

Events and directives: Setting a timer
Recognize event
Speak directive
PCM
10 minutes starting now.
MP3
Alexa, set a timer for 10 minutes.

Events and directives: Setting a timer
Recognize event
Speak directive
SpeechStarted event
SpeechFinished event
SetAlertSucceeded event
AVS controller
AudioPlayer
Alert manager
PCM
SetAlert directive
Alert
store
AlertStarted event
AlertEnteredForeground event
Time passes…
Local management
10 minutes starting now.
MP3
Alexa, set a timer for 10 minutes.

What do events and directives look like?
Alexa, what time is it?Alexa, what time is it?

SpeechRecognizer.Recognize
2017-10-10 18:19:25
{
"event": {
"header": {
"dialogRequestId": "dialogRequestId-2fa1867c-bfe2-4113-9be0-325a1c466156",
"namespace": "SpeechRecognizer",
"name": "Recognize",
"messageId": "e4717a2e-16bb-4153-ae31-3068107542be"
},
"payload": {
"profile": "CLOSE_TALK",
"format": "AUDIO_L16_RATE_16000_CHANNELS_1"
}
},
"context": [
{
"header": {
"namespace": "AudioPlayer",
"name": "PlaybackState"
},
"payload": {
"token": "",
"playerActivity": "IDLE",
"offsetInMilliseconds": "0"
}
Event
Alexa, what time is it?

2017-10-10 18:19:25
{
"event": {
"header": {
},
"payload": {
}
},
"context": [
{
"header": {
},
"payload": {
"token": "",
}
Speaker.SetMute
2017-10-10 18:19:27
{
"directive": {
"header": {
"namespace": "Speaker",
"name": "SetMute",
"messageId": "27448793-7fc8-43bd-bed8-64d2e226717f"
},
"payload": {
"mute": false} } }
Directive
Event

Event
2017-10-10 18:19:25
{
"event": {
"header": {
},
"payload": {
}
},
"context": [
{
"header": {
},
"payload": {
"token": "",
}
Speaker.SetMute
2017-10-10 18:19:27
{
"directive": {
"header": {
"namespace": "Speaker",
"name": "SetMute",
"messageId": "27448793-7fc8-43bd-bed8-64d2e226717f"
},
"payload": {
"mute": false} } }
Directive
SpeechSynthesizer.Speak
2017-10-10 18:19:27
{
"directive": {
"header": {
"dialogRequestId": "dialogRequestId-2fa1867c-bfe2-4113-9be0-
325a1c466156",
"namespace": "SpeechSynthesizer",
"name": "Speak",
"messageId": "2cc2674d-04b6-44c3-aa89-4d06b065930c"
},
"payload": {
"url": "cid:DeviceTTSRenderer_670aa604-26ee-4cff-a053-
c33604bf68a4_1370789680",
"token": "amzn1.as-
ct.v1.Domain:Application:Notifications#ACRI#DeviceTTSRenderer_670aa604-
26ee-4cff-a053-c33604bf68a4",
"format": "AUDIO_MPEG"
}
Directive
It’s 1:15 p.m.

Introducing the AVS device SDK
• Develop commercial-grade device software for AVS products
• Provides C++-based libraries to:
• Process audio inputs
• Establish persistent connections with AVS
• Handle all Alexa interactions on the device

AVS device SDK architecture
1. Audio input processor (AIP)
2. Wake word detection
3. Alexa communications library (ACL)
4. Alexa directive sequencer library (ADSL)
5. Capability agents
6. Activity focus manager library (AFML)

Capture and process audio
Shared data stream
• Pulls in audio data onto local device memory
Wake word engine
• Constantly examines the data in the shared data stream
• Compares it to onboard model for “Alexa” or other wake word
Audio input processor
• Sends audio data only when triggered
• Will continue to send audio until it receives a stop directive

Exchange information with AVS
• ACL forwards directives to the ADSL
• Initiates a persistent downchannel stream
• Opens and closes bidirectional event streams

Exchange information with AVS
HTTP/2 persistent connection
Event-specific stream
Downchannel stream Notifications
App-initiated actions
Recurring alerts
Music request
event
Play directive
Client device

Deliver the instructions
Alexa directive
sequencer library
Directives
Alexa
communication
library
• Accepts directives from the ACL
• Sequences directives in correct order
• Forwards directives to their
appropriate capability agents

Capability
agent
AudioPlayer
Alexa directive
sequencer library
Play Directive

Capability
agent
Capability
agent
AlertsAudioPlayer
Alexa directive
sequencer library
Set Timer Directive

Capability
agent
Capability
agent
Capability
agent
SpeechSynthesizerAlertsAudioPlayer
Alexa directive
sequencer library
Speech Directive

Prioritize user interactions
• The AFML controls which capability agents
may use the media player at any given time.

Alexa focus
manager library

Channel priority Capability agents
1. Dialog SpeechRecognizer,
SpeechSynthesizer
2. Alerts Alerts
3. Content AudioPlayer
Alexa focus
manager library

Perform specific functions
SpeechRecognizer
Speech capture
SpeechSynthesizer
Alexa speech output
AudioPlayer
Control music playback
Alerts
Timers and alarms

Perform specific functions
Notifications
Cloud-initiated messages
System
Communicate status to AVS
PlaybackController
Start/stop/skip track using buttons
TemplateRuntime
Render visual metadata
Speaker
Volume control (incl. mute)

AVS device SDK interaction
Alexa, what’s today’s weather?
“Alexa”
Automated speech
recognition (ASR)
Natural language
understanding (NLU)
Text to speech (TTS)
Skills
Weather
Recognize intent
AVS capability
Stream text/URL
SDK module (source)
Audio signal processor Media player
Wake word
detection
Wake word engine
Shared data
stream
Audio input
processor
Alexa
communication
library
Alexa directive
sequencer
library
Alexa focus
manager library
Capability
agent
External interfaceAlexa Voice Service Client (binary)
It’s 60 degrees and raining in Seattle.
Event
Recognize speech
Music
service
Directive
Alexa Voice Service

AVS device SDK is…
Portable - Extensible - Modular - Accessible

• Set up the SDK on Linux, MacOS, or Raspberry Pi
• Extend to many hardware platforms
• Garmin cross-compiled the SDK to Android OS
AVS Device SDK is…

• Add new features with a minimum of glue
code
• Hisense wrote their own capability agent to
implement display cards.

• Use libraries you need, discard what you don’t
• Choose your own wake word engine
• Sonos used their own media player.

• SDK C++ source code available on GitHub—no
cost to experiment, prototype, and build
• Modify behaviors to enhance product-specific UX

Prototype with the
AVS Device SDK sample app
• Use a commercial-grade SoC
or Raspberry Pi
• Hands-free implementation
with “Alexa” wake word
detection NXP PICO-i.MX7D Raspberry Pi

Build the AVS device SDK
Open your workshop kit and power up your Raspberry Pi!
Follow the instructions in your lab manual at:
https://avs-sdk-workshop.github.io/
W o r k s h o p a c t i v i t y :

Thank you!
I n t e g r a t e A l e x a i n t o Y o u r
P r o d u c t U s i n g t h e A V S
D e v i c e S D K
Visit the Alexa developer portal
for more information:
Documentation, tutorials ,webinars, and more
https://developer.amazon.com/alexa
AVS: Alexa Voice Service
https://developer.amazon.com/avs
ASK: Create custom skills for Alexa
https://developer.amazon.com/ask
Smart home:
https://developer.amazon.com/alexa/smart-
home

ALX307_Integrate Alexa into Your Product

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to ALX307_Integrate Alexa into Your Product

Similar to ALX307_Integrate Alexa into Your Product (20)

More from Amazon Web Services

More from Amazon Web Services (20)

ALX307_Integrate Alexa into Your Product