ALX202_Integrate Alexa voice technology into your product with the Alexa Voice Service

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Integrate Alexa voice technology into
your product with the Alexa Voice
Service

Hello, my name is…
Travis Grizzel
Principal Solutions Architect
Alexa Voice Service
grizzel@amazon.com

Voice is the future

• Single microphone
• Always on
The evolution of voice interaction

• Always on
• Push to talk
• Built-in speaker

• Always on
• Push to talk
• Voice activated
• Noise cancellation

• Always on
• Push to talk
• Voice Activated
• Noise cancellation
• 360 degree mic coverage
• Increased detection range

Amazon Echo: The 1st Alexa-enabled device
“Amazon’s Echo might be the
most important product in years”
“Amazon Echo proves the viability
of voice-based home computing.”
“Amazon Alexa just conquered CES 2017.
The world is next.”
“Alexa is clearly the star of IFA 2017”

Hi Alexa… Hello!

Smart Home
Control
Timers and Alarms
Cloud-Based
Intelligence
Shopping
Music
News and
Information
Custom Skills
Always getting smarter

Tens of Thousands
Alexa Developers
25,000+
Alexa Skills
1,000+
Works With Amazon Alexa
Certified Products
Tens of Millions
Alexa-Enabled Devices

Amazon Alexa-enabled products

Vision: Alexa anywhere
Home Audio
Thermostats
Automobiles
Maker boards
Wearables
Small Appliances
Smartphones
Large AppliancesHome Security
TV / Video

Alexa-enabled products

Cloud-Based Intelligence
Voice Design Made Simple
Easy to Integrate
Free to Use
Tap Into a Growing Network
Reach More Customers
Alexa Voice Service

Directives
Instructions sent from AVS to Client
Events
Information sent from Client to
AVS
Voice Hardware
Audio Front End
Device Software
AVS Device SDK
Simplifying voice interaction

“Alexa, what time is it?”
Wake Word
Engine
“Play” Directive
Intent
Recognition Result
Recognition Result
Text/SSML
Speech
Platform
Skills
Music
Service
ASR
NLU
TTS
“Recognize” Event
Streaming URL
User’s Utterance
Alexa Voice Response
Intent
Alexa Voice Service
Speech
Capture via
Microphone
Play back
through
Speaker

What do Events and Directives look like?
Alexa, what time is it?Alexa, what time is it?

SpeechRecognizer.Recognize
2017-10-10 18:19:25
{
"event": {
"header": {
"dialogRequestId": "dialogRequestId-2fa1867c-bfe2-4113-9be0-325a1c466156",
"namespace": "SpeechRecognizer",
"name": "Recognize",
"messageId": "e4717a2e-16bb-4153-ae31-3068107542be"
},
"payload": {
"profile": "CLOSE_TALK",
"format": "AUDIO_L16_RATE_16000_CHANNELS_1"
}
},
"context": [
{
"header": {
"namespace": "AudioPlayer",
"name": "PlaybackState"
},
"payload": {
"token": "",
"playerActivity": "IDLE",
"offsetInMilliseconds": "0"
}
Event
Alexa, what time is it?

Event
SpeechRecognizer.Recognize
2017-10-10 18:19:25
{
"event": {
"header": {
"dialogRequestId": "dialogRequestId-2fa1867c-bfe2-4113-9be0-325a1c466156",
"namespace": "SpeechRecognizer",
"name": "Recognize",
"messageId": "e4717a2e-16bb-4153-ae31-3068107542be"
},
"payload": {
"profile": "CLOSE_TALK",
"format": "AUDIO_L16_RATE_16000_CHANNELS_1"
}
},
"context": [
{
"header": {
"namespace": "AudioPlayer",
"name": "PlaybackState"
},
"payload": {
"token": "",
"playerActivity": "IDLE",
"offsetInMilliseconds": "0"
}
Alexa, what time is it?
Directive
SpeechSynthesizer.Speak
2017-10-10 18:19:27
{
"directive": {
"header": {
"dialogRequestId": "dialogRequestId-2fa1867c-bfe2-4113-9be0-
325a1c466156",
"namespace": "SpeechSynthesizer",
"name": "Speak",
"messageId": "2cc2674d-04b6-44c3-aa89-4d06b065930c"
},
"payload": {
"url": "cid:DeviceTTSRenderer_670aa604-26ee-4cff-a053-
c33604bf68a4_1370789680",
"token": "amzn1.as-
ct.v1.Domain:Application:Notifications#ACRI#DeviceTTSRenderer_670aa604-
26ee-4cff-a053-c33604bf68a4",
"format": "AUDIO_MPEG"
}
Directive
It’s 1:15 PM.

Events and Directives: Setting a timer
Recognize event
Speak directive
PCM
10 minutes starting now.
MP3
Alexa, set a timer for ten minutes.

Events and Directives: Setting a timer
Recognize event
Speak directive
SpeechStarted event
SpeechFinished event
SetAlertSucceeded event
AVS Controller
AudioPlayer
Alert Manager
PCM
SetAlert directive
Alert
Store
AlertStarted event
AlertEnteredForeground event
Time passes….
Local management
10 minutes starting now.
MP3
Alexa, set a timer for ten minutes.

Exchange information with AVS
HTTP/2 persistent connection
Event-specific Stream
Downchannel Stream Notifications
App-initiated actions
Recurring alerts
Music Request
Event
Audio Play Directive
Client device

Introducing the AVS Device SDK
• Develop commercial-grade device software for AVS products
• Provides C++-based libraries to:
• Process audio inputs
• Establish persistent connections with AVS
• Handle all Alexa interactions on the device

AVS Device SDK architecture
1. Audio Input Processor (AIP)
2. Wake Word Detection
3. Alexa Communications Library (ACL)
4. Alexa Directive Sequencer Library (ADSL)
5. Capability Agents
6. Activity Focus Manager Library (AFML)

Capture and process audio
Shared Data Stream
• Pulls in audio data onto local device memory
Wake Word Engine
• Constantly examines the data in the Shared Data Stream
• Compares it to onboard model for “Alexa”
Audio Input Processor
• Sends audio data only when triggered
• Will continue to send audio until it receives a stop directive

Exchange information with AVS
• Initiates a persistent downchannel stream
• Opens and closes bi-directional event streams
• ACL forwards directives to the ADSL

Deliver the instructions
Alexa Directive
Sequencer Library
Directives
Alexa
Communication
Library
• Accepts directives from the ACL
• Sequences directives in correct order
• Forwards directives to their
appropriate Capability Agents

Capability
Agent
AudioPlayer
Alexa Directive
Sequencer Library

Capability
Agent
Capability
Agent
AlertsAudioPlayer
Alexa Directive
Sequencer Library

Capability
Agent
Capability
Agent
Capability
Agent
SpeechSynthesizerAlertsAudioPlayer
Alexa Directive
Sequencer Library

Prioritize user interactions
• The AFML controls which Capability Agents
may use the media player at any given time

Alexa Focus
Manager Library

Channel priority Capability Agents
1. Dialog SpeechRecognizer,
SpeechSynthesizer
2. Alerts Alerts
3. Content AudioPlayer
Alexa Focus
Manager Library

Perform specific functions
SpeechRecognizer
Speech capture
SpeechSynthesizer
Alexa speech output
AudioPlayer
Control music playback
Alerts
Timers and Alarms

Perform specific functions
Notifications
Cloud-initiated messages
System
Communicate status to AVS
PlaybackController
Start/stop/skip track using buttons
TemplateRuntime
Render visual metadata
Speaker
Volume control (incl. mute)

AVS Device SDK interaction
Alexa, what’s today’s weather?
“Alexa”
Automated Speech
Recognition (ASR)
Natural Language
Understanding (NLU)
Text to Speech (TTS)
Skills
Weather
Recognize Intent
AVS Capability
Stream Text/URL
SDK Module (Source)
Audio Signal Processor Media Player
Wake Word
Detection
Wake Word Engine
Shared Data
Stream
Audio Input
Processor
Alexa
Communication
Library
Alexa Directive
Sequencer
Library
Alexa Focus
Manager Library
Capability
Agent
External InterfaceAlexa Voice Service Client (Binary)
It’s 60 degrees and raining in Seattle
Event
Recognize Speech
Music
Service
Directive
Alexa Voice Service

AVS Device SDK is…
Portable - Extensible - Modular - Accessible

• Set up the SDK on Linux, MacOS, or Raspberry Pi
• Extend to many hardware platforms
• Garmin cross-compiled the SDK to Android OS

• Add new features with a minimum of glue
code
• Hisense wrote their own capability agent to
implement Display Cards

• Use libraries you need, discard what you don’t
• Choose your own Wake Word Engine
• Sonos used their own Media Player

• SDK C++ source code available on GitHub—no
cost to experiment, prototype, and build
• Modify behaviors to enhance product-specific UX

Prototype with the
AVS Device SDK sample app
• Use a commercial-grade SoC
or Raspberry Pi
• Hands-Free implementation
with “Alexa” wake word
detection NXP PICO-i.MX7D Raspberry Pi

What about the hardware?

How do you build Alexa into your product?

White Box Design End Device
Continuum of reference designs
Goertek Puck Your Product Here

Form Factor
Reference
Design
Goertek Puck Your Product HereGemtek Vita FFRD

AVS-Enabled
Module
Form Factor
Reference
Design
Goertek Puck
Systems Integrator
AVS Module
Gemtek Vita FFRD Your Product Here

Development Kit AVS-Enabled
Module
Form Factor
Reference
Design
Conexant 4 mic
Dev Kit
Goertek Puck Your Product HereGemtek Vita FFRD
Systems Integrator
AVS Module

Touch Activated
Interaction models—introducing terms

Hands-Free
Voice Activated

Far-Field
Voice Activated

Microphone Array
ADC ADC ADC ADC
Hardware architecture—what’s in an AVS product?

Hardware architecture—what’s in an AVS client?
DSP
Amp DAC
Audio
Algorithms
Microphone Array
ADC ADC ADC ADC

DSP
Amp DAC
Audio
Algorithms
SoC
Wifi / Ethernet
AVS Client
WWE
Microphone Array
ADC ADC ADC ADC
Audio
Drivers
Hardware architecture—what’s in an AVS client?

Choosing the right audio front end
Number of
Microphones
Microphone
Arrangement
Audio Algorithms

Audio algorithms
Wake Word Engine: identifies “Alexa” wake word to start sending speech to cloud
• Model size:
Bigger is usually better
Requires more memory on client
• Cloud-based verification:
AVS double-checks your client’s WWE

Audio algorithms
ResponseAccuracy
Far Field Hands Free
SNR
Noise Reduction: remove background noise for clear speech capture
SNR = Signal to Noise Ratio

Audio algorithms
Acoustic Echo Cancellation: use audio feedback to cancel echo and improve SNR
Music playing
on device
Speaker output
Voice of User
Microphone
on device
DSP
Audio
Algorithms

First Last
Beamforming: locate the source of speech and pick it out of background noise
Audio algorithms

Audio algorithms
First Last
Beamforming: locate the source of speech and pick it out of background noise

Audio algorithms
Beamforming: orientation matters!
90°
90°
15°
• Circular arrays work best at <15° angle, such as on a table
• Wall or ceiling mounting may negatively affect beamforming

Microphone arrangement
Linear Array for
Wall-Mounted
Circular Array
for Tabletop

Number of microphones
1 Mic for
Touch Activated
2+ Mic for Hands-Free
and Far-Field

Number of microphonesResponseAccuracy
Far Field – 7 mic
SNR
Far Field – 4 mic
Hands Free – 2 mic
• 2-mic dev kits perform comparably
• More mics is usually better, but have
diminishing returns
• In high SNR, fewer mics works well
enough
• As SNR decreases, even 7-mic dev
kits eventually lose accuracy

• Integrate high-performance voice capture
• Dev kits include mic arrays, DSPs, SoCs
• Supported by leading chipset providers
Development kits for AVS

What’s included in a development kit?
DSP Board and/or
SoC
AVS Sample Client
Mic Array Board
Cables and Power
Adapter
Audio Processing Algorithms,
WWE, and Software Drivers
Documentation, Collaterals,
and Software Tools

INTEL
8-Mic Circular Array
XMOS
4-Mic Linear Array
AMAZON
7-Mic Circular Array
CONEXANT
4-Mic Square Array
CIRRUS LOGIC
2-Mic Linear Array
MICROSEMI
2-Mic Linear Array
CONEXANT
2-Mic Linear Array
AVS development kits

Bringing it all together
• Voice will fundamentally change the way people interact with technology
• Vision—Alexa is ubiquitous
• Alexa offers users the ability to interact with your device in new ways
• A quality Alexa implementation requires careful consideration around
hardware and software
• We’re making this easier for companies by offering development kits and
an SDK

My recommendations
• Build something unique
• Start with the user experience
• 7+ mics may not be necessary—align to the anticipated use case
• Early industrial design decisions will heavily impact the quality of the
Alexa experience
• Dev kits, dev kits, dev kits

My recommendations (cont.)
• Leverage the AVS SDK
• Give plenty of time
• Leave headroom for new Alexa capabilities
• Be prepared for AVS certification
• Engage with the AVS team
• Leverage solutions integrators

What will you create?
developer.amazon.com/alexa-voice-service/build

Thank you!
I n t e g r a t e A l e x a v o i c e
t e c h n o l o g y i n t o y o u r
p r o d u c t w i t h t h e A l e x a
V o i c e S e r v i c e
Visit the Alexa Developer Portal
for more information:
Documentation, Tutorials, Webinars, and more
developer.amazon.com/alexa
AVS: Alexa Voice Services
developer.amazon.com/avs
ASK: Create custom skills for Alexa
developer.amazon.com/ask
Smart Home:
developer.amazon.com/alexa/smart-home

ALX202_Integrate Alexa voice technology into your product with the Alexa Voice Service

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to ALX202_Integrate Alexa voice technology into your product with the Alexa Voice Service

Similar to ALX202_Integrate Alexa voice technology into your product with the Alexa Voice Service (20)

More from Amazon Web Services

More from Amazon Web Services (20)

ALX202_Integrate Alexa voice technology into your product with the Alexa Voice Service