Voice recognition is a natural method that people can use to interact with and automate smart devices. In this session, we build a microservice for automation of IoT using local fog computing resources and cloud-based serverless functions. We also create a voice-enabled chatbot that triggers automatic actions on an IoT network.

1. Voice Assistants and
IoT Automation
Intel, Software and Services Group
Daniel Holmlund - @agnathan

2. Voice Assistants
And
IoT Automation
Agenda
➔ Overview of Cloud and Edge Components needed to enable Voice IoT
Automation
➔ Installing Voice Assistants on the Intel® IoT Gateway
◆ Amazon Alexa*, Google Assistant* and Mycroft.ai*
➔ Rendering Voice Data into Intent
◆ Alexa Voice Service*, Google Actions*, Mycroft Intent*
➔ Intent Fulfillment
◆ Amazon Lambda*, Google Cloud Functions*
◆ Building Our Own Intent Fulfilment Service on the Intel* IoT Gateway
➔ Build Our Own IoT Automation Service on the Intel* IoT Gateway
◆ Discussion about State, Device Shadows, Flux Pattern

3. Voice and Automation Overview Diagram
Google*
Actions
Alexa* Voice
Services
Google*
Cloud Functions
Amazon* Lambda
Voice Fulfillment
Service
Edge
Network
Cloud
Automation Service
Intel IoT Gateway
* Voice Agents are
installed on the
Gateway
Google Assistant*
Amazon Alexa*
Mycroft.ai*
A voice to IoT device
automation workflow can
be built in the cloud or it
can be built entirely on the
edge network.

4. Questions to
Consider
➔ Do I want to enable every IoT Device with a Voice Assistant or just
a Gateway?
➔ How do I represent the state of an IoT device? How do I change the
state of an IoT device with voice?
➔ How do I build a voice to IoT automation workflow in the cloud?
➔ How do I build a voice to IoT automation workflow on the local
edge network?
➔ What are the tradeoffs building in the cloud vs on the edge?
➔ How can I build an IoT Automation service?

5. Installing Voice Assistants on the
Intel®
IoT Gateway

6. Demo: Voice Assistants Installed on Intel®
IoT Gateway

7. Alexa Voice Service* Installation
➔ Alexa Voice Service* allows you
to install the Alexa voice
assistant on a device.
➔ Alexa Skill Kit* enables you to
write a program that enables
Alexa Voice Service to respond to
new voice commands or
questions.

8. Register a New Device
To Create a New Device on the Alexa Voice
Service* enter the following:
➔ Device Information - Create an
identity for your Alexa* enabled
device.
➔ Security Profile - how Amazon*
identifies your device. This binds
your security profile to an oauth2 id
and secret
◆ Security Profile ID
◆ Client ID
◆ Client Secret
➔ Device Details - Image, Category, Is it
going to be a publically sold device,
etc ...

9. Amazon Alexa*
Installation on the
Intel® IoT Gateway
Amazon Provides Access to the Alexa Voice
Assistant* through Amazon Voice Services*
● Register for an Amazon Developer Account
○ Create an Alexa Voice Service Project
○ Create a device and security profile
■ ProductID (Device Type ID),
■ ClientID
■ ClientSecret
● Setup on the Intel IoT Gateway
○ git clone https://github.com/alexa/alexa-avs-sample-app.git
○ Enter the ProductID, ClientID and ClientSecret in the install script
○ Install VLC for audio API dependencies
○ Run generate.sh to create self-signed certificates
○ Run the sample Java application that communicates to AVS
○ Run the NodeJS Companion Application which listens for a response
from AVS
https://developer.amazon.com/alexa-voice-service
Intel® IoT Gateway
* Voice Agents are
installed on the Gateway
Amazon Alexa*

10. Google Assistant*
Installation on the
Intel® IoT Gateway
● Setup on the Google Cloud Platform
○ Create a Google Developer Project
○ Enable the Google Assistant API on the project
○ Enable the Billing API on the project (set a budget)
○ Create an OAuth Client ID and download the client_secret file in
JSON format.
● Local Setup Option 1: Google Assistant library for Python
$ sudo apt-get install python3-dev python3-venv
$ python3 -m venv env
$ env/bin/python -m pip install --upgrade pip setuptools
$ source env/bin/activate
$ (env) python -m pip install --upgrade google-assistant-library
$ (env) python -m pip install --upgrade google-auth-oauthlib[tool]
$ google-oauthlib-tool --client-secrets /home/pi/client_secret_client-id.json
--scope https://www.googleapis.com/auth/assistant-sdk-prototype --save
--headless
$ google-assistant-demo
● Local Setup Option 2 - Google Assistant gRPC API - exposes a low
level API which directly manipulates the audio bytes of an Assistant request
and response
https://developers.google.com/assistant/sdk/prototype
/getting-started-pi-python/config-dev-project-and-accou
nt
Intel® IoT Gateway
* Voice Agents are
installed on the Gateway
Google Assistant*

11. Resolving Voice to Intent

12. Services that Render Voice to Intent
Google
Actions
Amazon
Alexa
Service
Edge
Network
Cloud
Intel® IoT Gateway
* Voice Agents are
installed on the
Gateway
Google Assistant*
Amazon Alexa*
Mycroft.ai*
➔ Mycroft.ai does not require a cloud component. Voice
Fulfillment can happen entirely on a local edge network.
➔ This does not mean that a developer can forego
encryption and good security practices.
➔ Both Amazon and Google require a cloud and local
software to enable their voice assistants.
➔ Secure communications must be setup correctly
between the cloud and the gateway.
➔ Local software is activated with a keyword, voice is
recorded and uploaded to a cloud service that converts
the voice to an API call.

13. Communication between Intel® IoT Gateway and Cloud Voice Services
Google
Actions
Amazon
Alexa
Service
Edge
Network
Cloud
Intel® IoT Gateway
* Voice Agents are
installed on the
Gateway
Google Assistant*
Amazon Alexa*
Mycroft.ai*
➔ Amazon uses HTTPSv2 to communicate audio data
➔ Google uses gRPC, open source high performance RPC
framework
➔ Both Amazon and Google use Oauth2 client ID and
secrets to authenticate and authorize a device to their
respective cloud platforms
➔ You must implement this oauth2 interactive or use their
client libraries to do so.
➔ An Audio file with the voice request is sent to the cloud.
➔ Google Actions and Alexa Voice Services use Artificial
Intelligence techniques to resolve the audio into an
Intent.
➔ Once resolved to an Intent, the Intent is forwarded to a
Intent Fulfillment Service
HTTPSv2
gRPC

14. Training a Cloud Service to Understand
Voice
➔ Voice data is transcribed to
textual data.
➔ The developer provides a set
of example phrases that
match each intent
➔ The Cloud Service trains a set
of classifiers to match
example phrases to intents
➔ Each Cloud Provider uses
their own combination of
deep learning neural
networks, voice models and
classifiers
Example Phrases Matching Intent
Please turn on the lights. smarthome.lights.brightness.on
Are the lights on in the family room? smarthome.lights.brightness.check
Please turn the brightness of the lights
to 50%.
smarthome.lights.brightness.set
Please turn the lights to soft white. smarthome.lights.color.set
Please turn the lights off. smarthome.lights.brightness.off
Set the lights to pink! smarthome.lights.color.set

15. What is an Intent?
● Has a set of examples phrases
that identify the intent
● May require a context to be set
before text can be match to it.
● May trigger an event.
● May be require

16. Training a Cloud Service to Understand
Voice
➔ Example Phrases are matched to
intents
➔ Each intent has a series of “Entities” or
“Slots” which are keywords or phrases
that are converted to parameters
➔ Voice Assistant may ask follow up
questions to get all the intent’s
required parameters
➔ Parameters are labeled in a JSON
object and sent to the Fulfillment
Service over HTTPS
Example Phrases Matching “Entities” / “Slots”
Please turn on the lights. Green words are $Device
Are the lights on in the family room? Blue indicates $Color
Please turn the brightness of the
lights to 50%%.
Pink matches $Room
Please turn the lights to soft white. Red shows a $State

17. Mycroft Adapt Intent Parser*
The Adapt Intent Parser is an open source software
library for converting natural language into machine
readable data structures. Adapt is lightweight and
streamlined and is designed to run on devices with
limited computing resources. Adapt takes in natural
language and outputs a data structure that includes
the intent, a match probability, a tagged list of
entities. The software was developed at Mycroft AI by
a team led by Sean Fitzgerald, formerly one of the
developers of both Siri and Amazon Echo.
➔ https://adapt.mycroft.ai/start/examples
➔ https://github.com/MycroftAI

18. Google* owns Api.ai* - a tool for building voice interactions

19. Alexa Skill* Development Demo

20. Intent Fulfillment
and Serverless Functions

21. Intent Fulfillment: Voice and Automation Overview Diagram
Google*
Actions
Alexa* Voice
Services
Google*
Cloud Functions
Amazon* Lambda
Voice Fulfillment
Service
Edge
Network
Cloud
Automation Service
Intel®
IoT Gateway
* Voice Agents are
installed on the
Gateway
Google Assistant*
Amazon Alexa*
Mycroft.ai*
➔ The Fulfillment stage can be
continued in the cloud or send back to
the Intel IoT Gateway on the local
edge network.
➔ In order for Google Actions or Alexa
Voice Services to contact your Voice
Fulfillment Service it must be
accessible on the Internet or over a
VPN.

22. Serverless

23. Serverless Documentation

24. Alexa Lambda*

25. Google Cloud
Functions*
Hello World Voice
Fulfillment Template

26. Microservice
Fulfillment
From Google Actions*, Alexa Voice
Service* or Mycroft.ai* an HTTPS
request is received by the Intel® IoT
Gateway
The fulfillment function will create an
JSON Object that represents the state
of the light bulb.
The JSON looks like:
{sensor_id: ‘lightbulb’, value: ’on’,
timestamp: Date.now()}
The JSON is published over MQTT-TLS
Need screenshot of my code

27. Let’s Build Our Own Automation
Service

28. Voice and Automation Overview Diagram
Google*
Actions
Alexa* Voice
Services
Google*
Cloud Functions
Amazon* Lambda
Voice Fulfillment
Service
Edge
Network
Cloud
Automation Service
Intel IoT Gateway
* Voice Agents are
installed on the
Gateway
Google Assistant
Amazon Alexa
Mycroft.ai
➔ The Fulfillment stage can be
continued in the cloud or send back to
the Intel IoT Gateway on the local
edge network.
➔ In order for Google Actions or Alexa
Voice Services to contact your Voice
Fulfillment Service it must be
accessible on the Internet or over a
VPN.

29. 29
Automation IP Network
Upstream connection
3G/4G/5G or Internet
Secure the hardware, operating system, data and network.
Automation
Data
Processing
Intel IoT Gateway
Sensor data and actuator commands
are communicated over MQTT-TLS
Mosquitto broker running MQTT-TLS
on the edge network on the Intel® IoT
Gateway.
Automation Rules are stored in a
MongoDB database on the Intel® IoT
Gateway
Local Fulfillment Functions emits a
JSON object that contains the state
of the IoT Device that the
automation.
ManagementSecurity

30. Administrative Interface

31. What is a Trigger?
A trigger is a predicate (condition)
function and an action function that
implements the rule based business
logic on the edge network.
The predicate function returns a true
or false, a “Go” or “No Go”
In our implementation, the stash
always holds the last sensor reading
from each sensor.

32. Device Triggers Actions
There are many possible events that an action function could trigger.
● Write Data to the LCD screen
● Restart the device
● Execute an application
● Run a script
● Request data item from the device
● Send an SMS messages to an administrator.
● Send time synchronization
● Send an alert to the cloud

33. Creating a Connection to the MQTT Broker
A connection URI looks like mqtt://192.168.1.1/ NodeJS has a NPM package for MQTT.
var mqtt = require(‘mqtt’) will import the
package
MQTT has an event driven loop that
allows you to customize your
response to different events.
On the Connect Event, we log that we
connected and subscribe to the
“announcement” topic
the “+” matches any string. We are
subscribing to all sensors that
33

34. The Context Object
This object holds
variables and libraries
that will be made
accessible to the
condition and trigger
functions.
If you want to use
HTTP or MQTT in a
trigger define them in
the context object.

35. Rules Engines
For each incoming piece of data
filter the list of rules by the sensor
that send the incoming data
and evaluate the condition function
with the context
if it returns true then execute the
trigger function

36. Condition Functions
Conditions Functions, also
called predicate functions,
always return TRUE or
FALSE.

37. Example MQTT Trigger Function
Example Trigger Function
Publish an JSON error
report to the MQTT topic
sensors/temperature/errors
mqttClient and ErrorModel
are part of the Context
Object and now are
accessible under the this
object
Create a new MongoDB
document called error.
Save error to the database
Be sure to put the
parentheses around the
entire function

38. Example HTTP
Trigger Function
Example Trigger Function
Create an HTTP POST request for the
Intel Edison with the LCD.
Send the HTTP request with the
this.http object.
Write two functions to handle the
success and error conditions.
Be sure to put the parentheses around
the entire function

39. References
● https://www.xda-developers.com/how-to-get-google-assistant-on-your-windows-mac-or-linux-machine/
● https://inteliot.withthebest.com/#/insights/59275dec9e27000004ac3eca
● https://en.wikipedia.org/wiki/Long_short-term_memory
● https://github.com/alexa/alexa-avs-sample-app
● https://github.com/alexa/alexa-avs-sample-app/wiki/Raspberry-Pi
● https://github.com/alexa/alexa-avs-sample-app/wiki/Create-Security-Profile

41. More about Voice
Recognizers
https://developer.amazon.com/public/
solutions/alexa/alexa-voice-service/re
ference/speechrecognizer#overview