SlideShare a Scribd company logo
1 of 49
Download to read offline
Developing Conversation
Actions for the Google
Assistant
+GreenIdo
@greenido
Developer Advocate
June 2017
Google Home — voice-activated speaker powered
The Google Assistant — A conversation between you and
Google that helps you get more done in your world.
Actions on Google — How developers can extend the
assistant (via Conversation Actions)
“Ok Google, talk to personal chef”
Conversation API, Actions SDK
Invoke “personal chef” action
“Sure, here’s personal chef.
Hi, I’m your personal chef, what
are you in the mood for?”
Speech to Text
“What protein would you
like to use?”
“Well, it’s kind of cold outside, so I’d like
something to …”
Text to Speech
“Sure, here’s your personal chef”
Speech to Text, NLP,
Knowledge Graph,
ML Ranking, User
Profile, Text to
Speech
Webhook
(optional)
“Ok Google, talk to personal chef”
api.ai
Invoke “personal chef” action
“Well, it’s kind of cold outside, so I’d like
something to …”
“Sure, here’s your personal chef”
Speech to Text, NLP,
Knowledge Graph,
ML Ranking, User
Profile, Text to
Speech
“Sure, here’s personal chef.
Hi, I’m your personal chef, what
are you in the mood for?”
“What protein would you
like to use?”
Speech to Text
Text to Speech
Intent
Triggered via a series of “user says” phrases
Share a common set of entities
- Create new Intent otherwise
Intent name
Name to use in API.AI console
Only used by the developer, not referenced in your code
Action name
String passed to the webhook that indicates what action is being performed
Can share the same action name across different Intents
API.AI specific use of “Action” compared to Google Assistant “Conversation Action”
Input and Output Contexts
You can require a context to be available before an Intent is enabled
Intent can set context to enable other Intents
Implement flow control between some Intents
Entities
Values that we are trying to capture from the user phrases
Can specify a parameter name and a type of value
Values can be optional
Values can be a list of fixed values
Text response
Spoken when the Intent is complete
Embed entity values if desired
- “I found a recipe for $protein and $vegetable!”
Webhook can dynamically adjust this response
Prompts
Specify follow up questions if a user doesn’t specify certain values
Read out in random order to make it more natural
Webhook with HTTPS POST request
Node.js client library
https://developers.google.com/actions/apiai/fulfillment
'use strict';
// Boilerplate setup
let ApiAiAssistant = require('actions-on-google').ApiAiAssistant;
let express = require('express');
let bodyParser = require('body-parser');
let app = express();
app.set('port', (process.env.PORT || 8080));
app.use(bodyParser.json({type: 'application/json'}));
// Create an instance of ApiAiAssistant
app.post('/', function (request, response) {
const assistant = new ApiAiAssistant(
{request: request, response: response});
});
// Create functions to handle requests here
// Start the server
let server = app.listen(app.get('port'), function () {
console.log('App listening on port %s', server.address().port);
console.log('Press Ctrl+C to quit.');
});
Node.js client library
https://developers.google.com/actions/apiai/fulfillment
const assistant = new ApiAiAssistant({request: request, response: response});
const WELCOME_INTENT = 'input.welcome'; // the action name from the API.AI intent
const NUMBER_INTENT = 'input.number'; // the action name from the API.AI intent
const NUMBER_ARGUMENT = 'input.mynum'; // the action name from the API.AI intent
function welcomeIntent (assistant) {
assistant.ask('Welcome to action snippets! Say a number.');
}
function numberIntent (assistant) {
let number = assistant.getArgument(NUMBER_ARGUMENT);
assistant.tell('You said ' + number);
}
let actionMap = new Map();
actionMap.set(WELCOME_INTENT, welcomeIntent);
actionMap.set(NUMBER_INTENT, numberIntent);
assistant.handleRequest(actionMap);
Facts About Google sample (context and deep links)
https://developers.google.com/actions/apiai/fulfillment
Requesting user’s name and location
https://developers.google.com/actions/assistant/helpers
Q: "Recommend me a local bookstore"
A: "To find bookstores near you, I'll just need to get your zip code from Google. Is that
okay?"
NAME (Registered device user’s full name)
DEVICE_PRECISE_LOCATION (Coordinates and street address)
DEVICE_COARSE_LOCATION (Zip code and city)
// assistant is an instance of ApiAiAssistant or ActionsSdkAssistant
let permission = assistant.SupportedPermissions.NAME;
assistant.askForPermission('To address you by name', permission);
Account linking with OAuth 2.0
https://developers.google.com/actions/identity/account-linking
"It looks like your <invocation name> account is not linked yet. You can link <invocation
name> to your Google Account from the Google Home app."
Google Home app provides a link to your login page.
OAuth token is presented to your webhook.
Will only connect to a login page on your domain, not a 3rd party login.
Speech Synthesis Markup Language (SSML) & Earcons
https://developers.google.com/actions/reference/ssml
<speak> <!-- Must be at the start of the string -->
<say-as interpet-as="characters">SSML</say-as>
<break time="3s"/>.
<audio src="https://example.com/file.mp3"></audio>
<say-as interpret-as="cardinal">10</say-as>.
<say-as interpret-as="ordinal">10</say-as>
<say-as interpret-as="characters">10</say-as>.
<sub alias="World Wide Web">WWW</sub>.
<p><s>This is one.</s><s>This is two.</s></p>
</speak> <!-- Must be at the end of the string -->
“S S M L”
[3 second pause]
[audio file plays]
“Ten”
“Tenth”
“One Oh”
World Wide Web
[two sentences]
g.co/actionswebsim
Action naming and policies
https://developers.google.com/actions/policies/general-policies
● Policies about what Conversation Actions are allowed
● See the guidelines for more information
● Invocation names must be unique since it is a global name
● Direct Actions not widely available to build and deploy,
register for interest
Conversation Design
GUI :: VUI What’s the difference?
Top 3 Design Tips
Create a
persona
1
Persona is conveyed
through:
● Tone
● Word and phrase choices
● Functional design
● Style
● Technique
● Voice
...and based on:
1. Your user population and
their needs
2. The imagery & qualities
associated with your
brand
Preserve and reinforce your
persona by engaging the
user as a separate entity
from the Google Assistant.
Own it.
Hey! This is ___
Welcome to ___
Ready to play ____
Hi! ___ here.
Hello. I’m ___
Greetings, human.
Welcome back to ___
Hey again. ___
Let’s play ___
Here’s your ___
Brought to you by ___
Hi there, ___
Let’s get started.
Ready for your ___
___, here to…
Live from ___
This is ___
What’s up, ___
and more...
Think outside
the box.
Literally.
2
Confidential and proprietary
code code code code code
code code code code
code code “ dialog string” code
code code code
code code code code code
code “dialog string” code code
code code code code code code code
code code code
code code
code
code code
code
code
code code code
code “dialog string” code code code
code code
code code code
code code code code code code code code
code code code code code code code code
code
STRUCTURE + CODE
Confidential and proprietary
DIALOG + STRUCTURE
In conversations,
there are no
“errors”.
3
real life
user
conditions{
“I don’t understand you”
When a so-called “error” occurs in a
conversation, it should be treated simply as
a new turn in the dialog, only with different
conditions.
User’s have intent
g.co/dev/ActionsDesign
Actions on Google Developer
Community
https://g.co/actionsdev
Actions on Google Developers
https://developers.google.com/actions
@ActionsOnGoogle
Thank you!
+GreenIdo
@greenido
Developer Advocate
June 2017

More Related Content

Viewers also liked

Google home and actions
Google home and actionsGoogle home and actions
Google home and actionsRafael Pacheco
 
Présentation Hardis groupe
Présentation Hardis groupePrésentation Hardis groupe
Présentation Hardis groupeHardis
 
Google home - Social media campaign - Pierre-clement Lihou
Google home - Social media campaign - Pierre-clement LihouGoogle home - Social media campaign - Pierre-clement Lihou
Google home - Social media campaign - Pierre-clement LihouPierre-Clement Lihou
 

Viewers also liked (8)

Google home and actions
Google home and actionsGoogle home and actions
Google home and actions
 
Présentation Hardis groupe
Présentation Hardis groupePrésentation Hardis groupe
Présentation Hardis groupe
 
Amazon Echo vs Google Home: The Smart Home Begins
Amazon Echo vs Google Home: The Smart Home BeginsAmazon Echo vs Google Home: The Smart Home Begins
Amazon Echo vs Google Home: The Smart Home Begins
 
Google Home Voice App
Google Home Voice AppGoogle Home Voice App
Google Home Voice App
 
Google home - Social media campaign - Pierre-clement Lihou
Google home - Social media campaign - Pierre-clement LihouGoogle home - Social media campaign - Pierre-clement Lihou
Google home - Social media campaign - Pierre-clement Lihou
 
Google Home
Google HomeGoogle Home
Google Home
 
Google home
Google homeGoogle home
Google home
 
Google Home and Google Assistant Workshop: Build your own serverless Action o...
Google Home and Google Assistant Workshop: Build your own serverless Action o...Google Home and Google Assistant Workshop: Build your own serverless Action o...
Google Home and Google Assistant Workshop: Build your own serverless Action o...
 

Similar to Developing Conversation Actions for the Google Assistant

Google Assistant - Why? How?
Google Assistant - Why? How?Google Assistant - Why? How?
Google Assistant - Why? How?Ido Green
 
Building Conversational Experiences with Actions on Google
Building Conversational Experiences with Actions on Google Building Conversational Experiences with Actions on Google
Building Conversational Experiences with Actions on Google Peter Friese
 
Building Conversational Experiences with Actions on Google
Building Conversational Experiences with Actions on GoogleBuilding Conversational Experiences with Actions on Google
Building Conversational Experiences with Actions on GooglePeter Friese
 
Esplorando Google Assistant e Dialogflow
Esplorando Google Assistant e DialogflowEsplorando Google Assistant e Dialogflow
Esplorando Google Assistant e DialogflowPaolo Montrasio
 
Construindo projetos para o Google Assistant - I/O 2019 Recap São Paulo
Construindo projetos para o Google Assistant - I/O 2019 Recap São PauloConstruindo projetos para o Google Assistant - I/O 2019 Recap São Paulo
Construindo projetos para o Google Assistant - I/O 2019 Recap São PauloAlvaro Viebrantz
 
Engage user with actions
Engage user with actionsEngage user with actions
Engage user with actionsMatteo Bonifazi
 
Hey hubballi! - Talk on "Actions on Google" #DevFestHubali
Hey hubballi! - Talk on "Actions on Google" #DevFestHubali Hey hubballi! - Talk on "Actions on Google" #DevFestHubali
Hey hubballi! - Talk on "Actions on Google" #DevFestHubali Smit Jethwa
 
How to Build a Dialogflow App by using Actions on Google.
How to Build a Dialogflow App by using Actions on Google.How to Build a Dialogflow App by using Actions on Google.
How to Build a Dialogflow App by using Actions on Google.Yuting Liu
 
Building google home applications tutorial
Building google home applications tutorialBuilding google home applications tutorial
Building google home applications tutorialAlex Austin
 
Open Social Introduction - JUG SummerCamp 2010
Open Social Introduction - JUG SummerCamp 2010Open Social Introduction - JUG SummerCamp 2010
Open Social Introduction - JUG SummerCamp 2010Tugdual Grall
 
Introduction to Web APIs and the Google+ API - BarCamp Phnom Penh 2011
Introduction to Web APIs and the Google+ API - BarCamp Phnom Penh 2011Introduction to Web APIs and the Google+ API - BarCamp Phnom Penh 2011
Introduction to Web APIs and the Google+ API - BarCamp Phnom Penh 2011traactivity
 
Jaoo - Open Social A Standard For The Social Web
Jaoo - Open Social A Standard For The Social WebJaoo - Open Social A Standard For The Social Web
Jaoo - Open Social A Standard For The Social WebPatrick Chanezon
 
Building Conversational Experiences for Google Assistant
Building Conversational Experiences for Google AssistantBuilding Conversational Experiences for Google Assistant
Building Conversational Experiences for Google AssistantNader Khaled
 
How to build virtual assistant like Jarvis (in Ironman) with Google Assistant...
How to build virtual assistant like Jarvis (in Ironman) with Google Assistant...How to build virtual assistant like Jarvis (in Ironman) with Google Assistant...
How to build virtual assistant like Jarvis (in Ironman) with Google Assistant...Tu Le Dinh
 
Final getting started with google actions
Final  getting started with google actionsFinal  getting started with google actions
Final getting started with google actionsAjaySingh1938
 
Exploring Google (Cloud) APIs with Python & JavaScript
Exploring Google (Cloud) APIs with Python & JavaScriptExploring Google (Cloud) APIs with Python & JavaScript
Exploring Google (Cloud) APIs with Python & JavaScriptwesley chun
 
Introduction to Cloud Computing with Google Cloud
Introduction to Cloud Computing with Google CloudIntroduction to Cloud Computing with Google Cloud
Introduction to Cloud Computing with Google Cloudwesley chun
 
OpenSocial Intro
OpenSocial IntroOpenSocial Intro
OpenSocial IntroPamela Fox
 

Similar to Developing Conversation Actions for the Google Assistant (20)

Google Assistant - Why? How?
Google Assistant - Why? How?Google Assistant - Why? How?
Google Assistant - Why? How?
 
Building Conversational Experiences with Actions on Google
Building Conversational Experiences with Actions on Google Building Conversational Experiences with Actions on Google
Building Conversational Experiences with Actions on Google
 
Building Conversational Experiences with Actions on Google
Building Conversational Experiences with Actions on GoogleBuilding Conversational Experiences with Actions on Google
Building Conversational Experiences with Actions on Google
 
Esplorando Google Assistant e Dialogflow
Esplorando Google Assistant e DialogflowEsplorando Google Assistant e Dialogflow
Esplorando Google Assistant e Dialogflow
 
Construindo projetos para o Google Assistant - I/O 2019 Recap São Paulo
Construindo projetos para o Google Assistant - I/O 2019 Recap São PauloConstruindo projetos para o Google Assistant - I/O 2019 Recap São Paulo
Construindo projetos para o Google Assistant - I/O 2019 Recap São Paulo
 
Engage user with actions
Engage user with actionsEngage user with actions
Engage user with actions
 
Hey hubballi! - Talk on "Actions on Google" #DevFestHubali
Hey hubballi! - Talk on "Actions on Google" #DevFestHubali Hey hubballi! - Talk on "Actions on Google" #DevFestHubali
Hey hubballi! - Talk on "Actions on Google" #DevFestHubali
 
How to Build a Dialogflow App by using Actions on Google.
How to Build a Dialogflow App by using Actions on Google.How to Build a Dialogflow App by using Actions on Google.
How to Build a Dialogflow App by using Actions on Google.
 
Action on Google
Action on GoogleAction on Google
Action on Google
 
Building google home applications tutorial
Building google home applications tutorialBuilding google home applications tutorial
Building google home applications tutorial
 
Open Social Introduction - JUG SummerCamp 2010
Open Social Introduction - JUG SummerCamp 2010Open Social Introduction - JUG SummerCamp 2010
Open Social Introduction - JUG SummerCamp 2010
 
Introduction to Web APIs and the Google+ API - BarCamp Phnom Penh 2011
Introduction to Web APIs and the Google+ API - BarCamp Phnom Penh 2011Introduction to Web APIs and the Google+ API - BarCamp Phnom Penh 2011
Introduction to Web APIs and the Google+ API - BarCamp Phnom Penh 2011
 
Jaoo - Open Social A Standard For The Social Web
Jaoo - Open Social A Standard For The Social WebJaoo - Open Social A Standard For The Social Web
Jaoo - Open Social A Standard For The Social Web
 
Google APPs and APIs
Google APPs and APIsGoogle APPs and APIs
Google APPs and APIs
 
Building Conversational Experiences for Google Assistant
Building Conversational Experiences for Google AssistantBuilding Conversational Experiences for Google Assistant
Building Conversational Experiences for Google Assistant
 
How to build virtual assistant like Jarvis (in Ironman) with Google Assistant...
How to build virtual assistant like Jarvis (in Ironman) with Google Assistant...How to build virtual assistant like Jarvis (in Ironman) with Google Assistant...
How to build virtual assistant like Jarvis (in Ironman) with Google Assistant...
 
Final getting started with google actions
Final  getting started with google actionsFinal  getting started with google actions
Final getting started with google actions
 
Exploring Google (Cloud) APIs with Python & JavaScript
Exploring Google (Cloud) APIs with Python & JavaScriptExploring Google (Cloud) APIs with Python & JavaScript
Exploring Google (Cloud) APIs with Python & JavaScript
 
Introduction to Cloud Computing with Google Cloud
Introduction to Cloud Computing with Google CloudIntroduction to Cloud Computing with Google Cloud
Introduction to Cloud Computing with Google Cloud
 
OpenSocial Intro
OpenSocial IntroOpenSocial Intro
OpenSocial Intro
 

More from Ido Green

How to get things done - Lessons from Yahoo, Google, Netflix and Meta
How to get things done - Lessons from Yahoo, Google, Netflix and Meta How to get things done - Lessons from Yahoo, Google, Netflix and Meta
How to get things done - Lessons from Yahoo, Google, Netflix and Meta Ido Green
 
Crypto 101 and a bit more [Sep-2022]
Crypto 101 and a bit more [Sep-2022]Crypto 101 and a bit more [Sep-2022]
Crypto 101 and a bit more [Sep-2022]Ido Green
 
The Future of Continuous Software Updates Is Here
The Future of Continuous Software Updates Is HereThe Future of Continuous Software Updates Is Here
The Future of Continuous Software Updates Is HereIdo Green
 
Open Source & DevOps Market trends - Open Core Summit
Open Source & DevOps Market trends - Open Core SummitOpen Source & DevOps Market trends - Open Core Summit
Open Source & DevOps Market trends - Open Core SummitIdo Green
 
DevOps as a competitive advantage
DevOps as a competitive advantageDevOps as a competitive advantage
DevOps as a competitive advantageIdo Green
 
Data Driven DevOps & Technologies (swampUP 2019 keynote)
Data Driven DevOps & Technologies (swampUP 2019 keynote)Data Driven DevOps & Technologies (swampUP 2019 keynote)
Data Driven DevOps & Technologies (swampUP 2019 keynote)Ido Green
 
Startups Best Practices
Startups Best PracticesStartups Best Practices
Startups Best PracticesIdo Green
 
Progressive Web Apps For Startups
Progressive Web Apps For StartupsProgressive Web Apps For Startups
Progressive Web Apps For StartupsIdo Green
 
Earn More Revenue With Firebase and AdMob
Earn More Revenue With Firebase and AdMobEarn More Revenue With Firebase and AdMob
Earn More Revenue With Firebase and AdMobIdo Green
 
How To Grow Your User Base?
How To Grow Your User Base?How To Grow Your User Base?
How To Grow Your User Base?Ido Green
 
Amp Overview #YGLF 2016
Amp Overview #YGLF 2016Amp Overview #YGLF 2016
Amp Overview #YGLF 2016Ido Green
 
AMP - Accelerated Mobile Pages
AMP - Accelerated Mobile PagesAMP - Accelerated Mobile Pages
AMP - Accelerated Mobile PagesIdo Green
 
From AMP to PWA
From AMP to PWAFrom AMP to PWA
From AMP to PWAIdo Green
 
Google Innovation 101
Google Innovation 101Google Innovation 101
Google Innovation 101Ido Green
 
סטארטאפ - איך? כמה? ולמה
סטארטאפ - איך? כמה? ולמהסטארטאפ - איך? כמה? ולמה
סטארטאפ - איך? כמה? ולמהIdo Green
 
איך להתחיל סטארטאפ 2016
איך להתחיל סטארטאפ 2016איך להתחיל סטארטאפ 2016
איך להתחיל סטארטאפ 2016Ido Green
 
Building a Progressive Web App
Building a  Progressive Web AppBuilding a  Progressive Web App
Building a Progressive Web AppIdo Green
 
How to start? - The Execution
How to start? - The Execution How to start? - The Execution
How to start? - The Execution Ido Green
 
How to make (more) money on the web?
How to make (more) money on the web?How to make (more) money on the web?
How to make (more) money on the web?Ido Green
 
How to start? (and excel with the elephants)
How to start? (and excel with the elephants)How to start? (and excel with the elephants)
How to start? (and excel with the elephants)Ido Green
 

More from Ido Green (20)

How to get things done - Lessons from Yahoo, Google, Netflix and Meta
How to get things done - Lessons from Yahoo, Google, Netflix and Meta How to get things done - Lessons from Yahoo, Google, Netflix and Meta
How to get things done - Lessons from Yahoo, Google, Netflix and Meta
 
Crypto 101 and a bit more [Sep-2022]
Crypto 101 and a bit more [Sep-2022]Crypto 101 and a bit more [Sep-2022]
Crypto 101 and a bit more [Sep-2022]
 
The Future of Continuous Software Updates Is Here
The Future of Continuous Software Updates Is HereThe Future of Continuous Software Updates Is Here
The Future of Continuous Software Updates Is Here
 
Open Source & DevOps Market trends - Open Core Summit
Open Source & DevOps Market trends - Open Core SummitOpen Source & DevOps Market trends - Open Core Summit
Open Source & DevOps Market trends - Open Core Summit
 
DevOps as a competitive advantage
DevOps as a competitive advantageDevOps as a competitive advantage
DevOps as a competitive advantage
 
Data Driven DevOps & Technologies (swampUP 2019 keynote)
Data Driven DevOps & Technologies (swampUP 2019 keynote)Data Driven DevOps & Technologies (swampUP 2019 keynote)
Data Driven DevOps & Technologies (swampUP 2019 keynote)
 
Startups Best Practices
Startups Best PracticesStartups Best Practices
Startups Best Practices
 
Progressive Web Apps For Startups
Progressive Web Apps For StartupsProgressive Web Apps For Startups
Progressive Web Apps For Startups
 
Earn More Revenue With Firebase and AdMob
Earn More Revenue With Firebase and AdMobEarn More Revenue With Firebase and AdMob
Earn More Revenue With Firebase and AdMob
 
How To Grow Your User Base?
How To Grow Your User Base?How To Grow Your User Base?
How To Grow Your User Base?
 
Amp Overview #YGLF 2016
Amp Overview #YGLF 2016Amp Overview #YGLF 2016
Amp Overview #YGLF 2016
 
AMP - Accelerated Mobile Pages
AMP - Accelerated Mobile PagesAMP - Accelerated Mobile Pages
AMP - Accelerated Mobile Pages
 
From AMP to PWA
From AMP to PWAFrom AMP to PWA
From AMP to PWA
 
Google Innovation 101
Google Innovation 101Google Innovation 101
Google Innovation 101
 
סטארטאפ - איך? כמה? ולמה
סטארטאפ - איך? כמה? ולמהסטארטאפ - איך? כמה? ולמה
סטארטאפ - איך? כמה? ולמה
 
איך להתחיל סטארטאפ 2016
איך להתחיל סטארטאפ 2016איך להתחיל סטארטאפ 2016
איך להתחיל סטארטאפ 2016
 
Building a Progressive Web App
Building a  Progressive Web AppBuilding a  Progressive Web App
Building a Progressive Web App
 
How to start? - The Execution
How to start? - The Execution How to start? - The Execution
How to start? - The Execution
 
How to make (more) money on the web?
How to make (more) money on the web?How to make (more) money on the web?
How to make (more) money on the web?
 
How to start? (and excel with the elephants)
How to start? (and excel with the elephants)How to start? (and excel with the elephants)
How to start? (and excel with the elephants)
 

Recently uploaded

Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructureitnewsafrica
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialJoão Esperancinha
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Nikki Chapple
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Nikki Chapple
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...BookNet Canada
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 

Recently uploaded (20)

Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorial
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
How Tech Giants Cut Corners to Harvest Data for A.I.
How Tech Giants Cut Corners to Harvest Data for A.I.How Tech Giants Cut Corners to Harvest Data for A.I.
How Tech Giants Cut Corners to Harvest Data for A.I.
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 

Developing Conversation Actions for the Google Assistant

  • 1. Developing Conversation Actions for the Google Assistant +GreenIdo @greenido Developer Advocate June 2017
  • 2.
  • 3. Google Home — voice-activated speaker powered The Google Assistant — A conversation between you and Google that helps you get more done in your world. Actions on Google — How developers can extend the assistant (via Conversation Actions)
  • 4.
  • 5.
  • 6. “Ok Google, talk to personal chef” Conversation API, Actions SDK Invoke “personal chef” action “Sure, here’s personal chef. Hi, I’m your personal chef, what are you in the mood for?” Speech to Text “What protein would you like to use?” “Well, it’s kind of cold outside, so I’d like something to …” Text to Speech “Sure, here’s your personal chef” Speech to Text, NLP, Knowledge Graph, ML Ranking, User Profile, Text to Speech
  • 7.
  • 8.
  • 9. Webhook (optional) “Ok Google, talk to personal chef” api.ai Invoke “personal chef” action “Well, it’s kind of cold outside, so I’d like something to …” “Sure, here’s your personal chef” Speech to Text, NLP, Knowledge Graph, ML Ranking, User Profile, Text to Speech “Sure, here’s personal chef. Hi, I’m your personal chef, what are you in the mood for?” “What protein would you like to use?” Speech to Text Text to Speech
  • 10.
  • 11.
  • 12. Intent Triggered via a series of “user says” phrases Share a common set of entities - Create new Intent otherwise
  • 13. Intent name Name to use in API.AI console Only used by the developer, not referenced in your code
  • 14. Action name String passed to the webhook that indicates what action is being performed Can share the same action name across different Intents API.AI specific use of “Action” compared to Google Assistant “Conversation Action”
  • 15. Input and Output Contexts You can require a context to be available before an Intent is enabled Intent can set context to enable other Intents Implement flow control between some Intents
  • 16. Entities Values that we are trying to capture from the user phrases Can specify a parameter name and a type of value Values can be optional Values can be a list of fixed values
  • 17. Text response Spoken when the Intent is complete Embed entity values if desired - “I found a recipe for $protein and $vegetable!” Webhook can dynamically adjust this response
  • 18.
  • 19.
  • 20. Prompts Specify follow up questions if a user doesn’t specify certain values Read out in random order to make it more natural
  • 21. Webhook with HTTPS POST request
  • 22.
  • 23. Node.js client library https://developers.google.com/actions/apiai/fulfillment 'use strict'; // Boilerplate setup let ApiAiAssistant = require('actions-on-google').ApiAiAssistant; let express = require('express'); let bodyParser = require('body-parser'); let app = express(); app.set('port', (process.env.PORT || 8080)); app.use(bodyParser.json({type: 'application/json'})); // Create an instance of ApiAiAssistant app.post('/', function (request, response) { const assistant = new ApiAiAssistant( {request: request, response: response}); }); // Create functions to handle requests here // Start the server let server = app.listen(app.get('port'), function () { console.log('App listening on port %s', server.address().port); console.log('Press Ctrl+C to quit.'); });
  • 24. Node.js client library https://developers.google.com/actions/apiai/fulfillment const assistant = new ApiAiAssistant({request: request, response: response}); const WELCOME_INTENT = 'input.welcome'; // the action name from the API.AI intent const NUMBER_INTENT = 'input.number'; // the action name from the API.AI intent const NUMBER_ARGUMENT = 'input.mynum'; // the action name from the API.AI intent function welcomeIntent (assistant) { assistant.ask('Welcome to action snippets! Say a number.'); } function numberIntent (assistant) { let number = assistant.getArgument(NUMBER_ARGUMENT); assistant.tell('You said ' + number); } let actionMap = new Map(); actionMap.set(WELCOME_INTENT, welcomeIntent); actionMap.set(NUMBER_INTENT, numberIntent); assistant.handleRequest(actionMap);
  • 25. Facts About Google sample (context and deep links) https://developers.google.com/actions/apiai/fulfillment
  • 26. Requesting user’s name and location https://developers.google.com/actions/assistant/helpers Q: "Recommend me a local bookstore" A: "To find bookstores near you, I'll just need to get your zip code from Google. Is that okay?" NAME (Registered device user’s full name) DEVICE_PRECISE_LOCATION (Coordinates and street address) DEVICE_COARSE_LOCATION (Zip code and city) // assistant is an instance of ApiAiAssistant or ActionsSdkAssistant let permission = assistant.SupportedPermissions.NAME; assistant.askForPermission('To address you by name', permission);
  • 27. Account linking with OAuth 2.0 https://developers.google.com/actions/identity/account-linking "It looks like your <invocation name> account is not linked yet. You can link <invocation name> to your Google Account from the Google Home app." Google Home app provides a link to your login page. OAuth token is presented to your webhook. Will only connect to a login page on your domain, not a 3rd party login.
  • 28. Speech Synthesis Markup Language (SSML) & Earcons https://developers.google.com/actions/reference/ssml <speak> <!-- Must be at the start of the string --> <say-as interpet-as="characters">SSML</say-as> <break time="3s"/>. <audio src="https://example.com/file.mp3"></audio> <say-as interpret-as="cardinal">10</say-as>. <say-as interpret-as="ordinal">10</say-as> <say-as interpret-as="characters">10</say-as>. <sub alias="World Wide Web">WWW</sub>. <p><s>This is one.</s><s>This is two.</s></p> </speak> <!-- Must be at the end of the string --> “S S M L” [3 second pause] [audio file plays] “Ten” “Tenth” “One Oh” World Wide Web [two sentences]
  • 30.
  • 31. Action naming and policies https://developers.google.com/actions/policies/general-policies ● Policies about what Conversation Actions are allowed ● See the guidelines for more information ● Invocation names must be unique since it is a global name ● Direct Actions not widely available to build and deploy, register for interest
  • 33.
  • 34. GUI :: VUI What’s the difference?
  • 35. Top 3 Design Tips
  • 37. Persona is conveyed through: ● Tone ● Word and phrase choices ● Functional design ● Style ● Technique ● Voice ...and based on: 1. Your user population and their needs 2. The imagery & qualities associated with your brand
  • 38. Preserve and reinforce your persona by engaging the user as a separate entity from the Google Assistant. Own it. Hey! This is ___ Welcome to ___ Ready to play ____ Hi! ___ here. Hello. I’m ___ Greetings, human. Welcome back to ___ Hey again. ___ Let’s play ___ Here’s your ___ Brought to you by ___ Hi there, ___ Let’s get started. Ready for your ___ ___, here to… Live from ___ This is ___ What’s up, ___ and more...
  • 40. Confidential and proprietary code code code code code code code code code code code “ dialog string” code code code code code code code code code code “dialog string” code code code code code code code code code code code code code code code code code code code code code code code “dialog string” code code code code code code code code code code code code code code code code code code code code code code code code code STRUCTURE + CODE
  • 42. In conversations, there are no “errors”. 3
  • 45. When a so-called “error” occurs in a conversation, it should be treated simply as a new turn in the dialog, only with different conditions.
  • 48. Actions on Google Developer Community https://g.co/actionsdev Actions on Google Developers https://developers.google.com/actions @ActionsOnGoogle