Recommended
More Related Content
Similar to Build Conversational Experiences with Actions on Google
Similar to Build Conversational Experiences with Actions on Google (20)
Recently uploaded
Recently uploaded (20)
Build Conversational Experiences with Actions on Google
- 1. Actions On Google Honey Sharma 🍯 TechnoJAM @honeycoder96 | honey.singhroi@gmail.com
- 3. What are AoG apps?
- 4. Learn how to build Actions for the Google Assistant that are useful for your community - whether it's a professional network, interest group, or your hometown.
- 5. Actions on Google Community Program developers.google.com/actions/community You are automatically invited to the program with publishing your first Action and will receive a Cool T Shirt + $200 Cloud Credits per month for a year The Assistant team also recognizes milestones: Traction Engagements Globalization
- 7. Action { conversation api request } { conversation api response } user input Action response
- 8. Ready to help across 500M+ devices
- 9. Devices with the Google Assistant built-in These include voice activated speakers like Google Home, Android phones, iPhones etc. The Google Assistant A conversation between you and Google that helps you get things done in your world. Actions on Google How developers can extend the assistant
- 11. JapaneseGerman FR French CA French KoreanUK English AU English CA English US English Italian Spanish BR Portuguese LATAM Spanish Hindi & IN English A growing audience: languages Swedish Danish Norwegian Indonesian Russian Thai Dutch
- 12. Hey Google, talk to __________. Hey Google, connect me with __________. Hey Google, get __________.
- 14. Ok Google, talk to Personal Chef Sure, here’s Personal Chef What are you in the mood for? What protein would you like to use? Speech to Text and Text to Speech Speech to Text Text to Speech .. . Invoke Personal Chef action Parse query and generate response Well, it’s kind of cold outside, so I’d like...
- 15. Well, it’s kind of cold outside, so I'd like something to warm me up, like a hot soup, and I want it fast.
- 16. Introduction to Dialogflow Darshan Baid | TechnoJam | @frunkad | its@darshan.ninja
- 17. “ Intent Matching — Match and categorize user utterances to an intent. Entity Extraction — Identify key words and phrases spoken by the user.@
- 19. Ok Google, talk to Personal Chef Sure, here’s Personal Chef What are you in the mood for? What protein would you like to use? Speech to Text NLP Knowledge Graph ML Ranking User Profile Speech to Text Text to Speech .. . .. . .. . Trigger WELCOME event .. . NLP: Intent Matching Entity Extraction Well, it’s kind of cold outside, so I’d like...
- 20. Entities: homemade cannoli “Find me a recipe for homemade cannoli” User Says Response Webhook “Find me a recipe for homemade cannoli” “Here’s the best one I can find”
- 22. Reach Users Darshan Baid | TechnoJAM | @frunkad | darshanbaid22@gmail.com
- 23. Discovery in the Google Assistant Directory
- 24. Ok Google, ask Personal Chef for a hot soup recipe Trigger Phrase Action Name Developer Specified Action Preposition Action Phrase Developer Specified Working Examples: Let’s speak to Domino’s Ask Dr. Doggy if dogs can eat chocolate Ok Google, let’s talk to Personal Chef Trigger Phrase Action Name Developer Specified In Dialogue Discovery: Explicit Triggering
- 25. In Dialogue Discovery: Implicit Triggering Hey Google, I want to work out. Hey Google, I want to play a game. Hey Google, what’s the surf report? Hey Google, tell me a joke.
- 26. Link to your Action from anywhere developers.google.com/actions/console/publishing#linking_to_your_actions
- 28. Speech Synthesis Markup Language (SSML) developers.google.com/actions/reference/ssml
- 31. Directory
- 32. Privacy Policy
- 33. Naming ✕ One-Word Names ✕ Common Phrases ✕ Generic Words ✕ Reserved words: OK, Google, Volume up, Game, Bot, Action, App
- 34. Console bit.ly/buildactions-console Build with Templates developers.google.com/actions/templates Build with Dialogflow developers.google.com/actions/dialogflow Build with Actions SDK developers.google.com/actions/sdk Resources to get started and support Documentation developers.google.com/actions Codelabs codelabs.developers.google.com Write and Deploy Functions bit.ly/firebase-get-started Deploy your fulfillment webhook using Cloud Functions for Firebase bit.ly/aog-cloud-functions
Editor's Notes
- Welcome the guests, share housekeeping rules and WiFi Password
- Welcome the guests, share housekeeping rules and WiFi Password
- Before we start, we want to make sure that everybody can feel safe - so this is our Code of Conduct. This should ensure for harassment-free and inclusive events. We’ve used here the standard Google Event Community Guidelines and Anti-Harassment Policy as guideline. Zero Tolerance Policy for harassment Including but not limited to: Stalking/following --- Offensive verbal language --- Deliberate intimidation --- Sexual imagery and language in public spaces --- Harassing photography or recording --- Inappropriate physical contact --- Sustained disruption of talks or other events --- Unwelcome sexual or physical attention --- Verbal language that reinforces social structures of domination In relation to, but not limited to: Neurodiversity --- Age --- Race --- Body size --- Color --- Disabilities --- National origin --- Appearance --- Gender identity --- Religion --- Gender expression --- Pregnancy --- Sexual orientation (https://sites.google.com/corp/google.com/devrelcon/community-guidelines)
- Finally, you can hyperlink to your Action from just about anywhere. You can share it in social media, promote through your own site or apps and encourage press to drive traffic to your Actions. If a user clicks on the link, they’ll be taken to a screen where they can read about your Action and try it out. https://developers.google.com/actions/console/publishing#linking_to_your_actions
- What is this event about? Part of a global event series that is supported by Google Developers and the Assistant Team (runs from now to July 15th) Idea behind it is to create Actions on Google for communities that you care about This can be a community around personal interests (developers, travellers, people around social causes) Local communities (the region, the country or the city) Business community (your own business, but also business information or product specifics)
- You are invited in the community with publishing your first Action. What does the program include besides a shirt and 200 USD Cloud Credits every month for a year? Invites to our community program's Google Group, Google homes, accelerated Actions review process & more! The Cloud Credits are also applicable to all Firebase Plans. Additionally, the team also supports you along the way with passing important milestones like getting initial traction, keeping users engaged or taking your Action global
- Speaker Intro Slides: 6 to 15 Time Limit for slides 7 to 12: 5mins. Won’t go deep
- One option is to use the Conversation API and the Actions SDK. Your Action receives a request containing the spoken text from the user as a string. Google handles the speech recognition for you. You parse the strings, and then generate a response, and Google handles speaking this back to the user. However, as we just mentioned, parsing natural language can be really difficult.
- At Google, we believe the future is AI first. We have been investing heavily in the areas of: Machine learning Speech recognition Language understanding These things come together in the Google Assistant. Allows you to have a conversation with Google, that helps you get things done. Because of these investments in AI, the conversation can be completely natural. Use your voice, ask in a natural way, and the Assistant helps you. It is already accessible on more than 500M devices across Android phones, the iPhone, Google Home and other voice-activated speakers, Android TVs, cars, Chromebooks, wearables and our messaging app Allo but it is soon coming to additional devices and contexts.
- So what does this mean for a developer? We have three parts in this new platform. The Google Assistant is the conversation between the user and Google. They can get things done by talking with the Assistant. There are many things users can do by just using the Assistant directly. And there are a lot of devices with the Google Assistant built-in These include voice activated speakers like Google Home, Android phones, iPhones etc. Actions on Google allows developers to extend the assistant. A developer can implement an Action. That is what we are going to focus on today.
- Today, Actions are available through the Assistant to users on several voice-enabled speakers like Google Home, Android phones, the iPhone, Android TVs, Chromebooks, headphones - and also cars. And the number of platforms and devices will continue to grow, meaning brands can reach a continuously expanding addressable audience and assist users across more contexts.
- Actions are also now available in many languages including US UK, Australian and Canadian English, French for France and Canada, German and Japanese, Italian, Spanish for Spain and Mexico, Portuguese Brazilian, Korean, Russian - with more languages coming this year.
- For users who already know your Action exists: they can engage with your Action by saying “Ok Google, talk to XXX”. Additionally, there are many more invocation paths to match how users may want to communicate to the Assistant that they want to engage with your Actions e.g. connect me with XXX, get XXX etc.
- Let’s now show a quick demo with something I call Personal Chef. This shows how an Assistant app is started, as well as a possible dialog with a user. Wayne: "Ok Google, let me talk to PERSONAL CHEF". Assistant: “Sure, here’s PERSONAL CHEF! <ding>” Action: "Hi, I’m your personal chef! What’re you in the mood for? Wayne: "Well, it’s kind of cold outside, so I'd like something to warm me up, like a hot soup, and I want it fast!" Action: "Alright, what protein would you like to use?" Wayne: "I have some chicken, and also canned tomatoes." Action: “Ok, well I think you should try the 'chicken tomato soup' recipe I found on example.com” Wayne: <looks at the camera with a smile> “Sounds good to me!” <STOP THE VIDEO HERE, ONLY SHOW THE DEMO!>
- So how does a Conversation Action work? The user needs to invoke your action. You say a phrase like “Ok Google, talk to personal chef”. This tells Google the name of the action to talk to. From this point onwards, the user is now talking to your conversation action. Your action generates dialog output, which then spoken to the user. The user then makes requests, your action processes it, and replies back again. The user has a two way dialog until the conversation is finished. We’ll talk more about the review process later, but user don’t need to pre-enable anything or install new apps to interact with your conversation actions.
- The demo conversation was actually quite complex. Our user said cold, warm and hot all in the same sentence, yet the app managed to capture the correct one. Could you imagine trying to write a regular expression or parser to try and extract meaning out of this? There are so many difficult cases that this is basically impossible for anything that is not trivial. So let’s see the various ways we could build this interaction.
- Speaker Intro Slides: 6 to 15 Time Limit for slides 7 to 12: 5mins. Won’t go deep
- Fortunately, Google provides tools that make handling this kind of thing really easy! Dialogflow is one of these. It’s a platform that makes it incredibly straightforward to build conversational experiences. You might not even have to write any code. We are going to give you an overview of it today, and it is what most of you should use for implementing your own Action
- So what is Dialogflow? It provides an intuitive graphical user interface to create conversational experiences. You program in, few example sentences of things that a user might speak. You can specify what values you need to get from the user. It then uses machine learning to understand the sentences and manage the conversation. The key part here is that you no longer need to process the raw strings - Dialogflow does this for you.
- So you can see here where Dialogflow fits in. It handles the conversation invocation and fulfillment just like we showed earlier.
- Dialogflow handles the conversation for you. Once the user is talking to your Action, they then start off with something Google calls the “user says” phrase. “Find me a recipe for homemade cannoli.” The Google Assistant and Dialogflow then process this, and find the appropriate intent to handle this phrase. The phrase is processed to extract entities, which are important pieces of information you are looking for. Then, it calls your webhook with these entities and the action name, and the webhook can then do something with this, and then generate a response that is spoken back to the user. You can also create your own demo here :)
- We would build our first really simple app - Introduce everyone to platform Give them 15 minutes to explore the dialogflow interface
- Speaker Intro Slides: 6 to 15 Time Limit for slides 7 to 12: 5mins. Won’t go deep
- There is also have a directory where users can explore the Actions that are on offer - on the phone and desktop Users don’t have to install or pay for anything - all of the Actions are available just by asking.
- The basic way to access your Action is invoking it by name. When you submit your Action, you can provide a set of trigger phrases that the Assistant will recognize as a request to speak to your Action. You can also support deep links, so users can ask your Action for a particular thing.
- In the long term, users will be able to find your Action based on its specific capabilities. If they ask to play a game, the Assistant will offer them a few options from the Actions available.
- Finally, you can hyperlink to your Action from just about anywhere. You can share it in social media, promote through your own site or apps and encourage press to drive traffic to your Actions. If a user clicks on the link, they’ll be taken to a screen where they can read about your Action and try it out. https://developers.google.com/actions/console/publishing#linking_to_your_actions
- Continue working on your Action
- Customize your action’s voice by adding SSML. You can add breaks, have it speak in cardinals, ordinals or in digits. You can add sounds and more! Demo of audio on https://developers.google.com/actions/reference/ssml
- Make your Action richer and enhance the user experience and increase the polish of your actions by adding a variety of sounds
- Before we begin, we shortly want to give you a short overview about how you make an App, that passes the review, as every app has to be submitted for review before publishing it. Tutorial: https://medium.com/google-developers/how-to-publish-an-assistant-app-that-will-pass-the-review-38a47ac64a0e You can also choose only show the video and not the explanation or the other way around
- One reason is: Incomplete store listing data You can see in the image below what are the top ‘missing parts’: The Action name Pronunciation — Picking a name with a recognizable pronunciation is critical. Invocation — How the users will call your Action. For example: “OK Google talk with bitcoin info”. Don’t forget you can mix images and diagrams as you see fit. The more graphical, the better! Make sure that the images are in high quality and the Action description is accurate. The Large banner image should be 1920 by 1080 and the Small square logo is 192 by 192.
- Another important part is your privacy policy. We frequently see example.com for the privacy policy which (of course) is not good. It must be public on the web and related to your Action. If you don’t have a web site, you can create one for free with google sites or other services. Here is one that Ido created for the Action “bitcoin Info” - feel free to copy it. https://sites.google.com/corp/view/bitcoin-info/
- One-word names are not allowed, unless the name is unique to your brand or trademark. A name uniquely identifies your Action, so it must distinguish itself from others and from features of the Assistant. You should avoid very common phrases like, thank you, good evening or OMG. Don’t use Generic words or phrases such as categories of products, services, or content. Travel or Sports for example. Think of i this way…If your name can describe a category of Actions rather than a specific function, change it to be more specific. Some words and phrases are reserved and cannot be used in names including: OK, Google, volume up, game, bot, action, and app. For example: “Talk to bitcoin info” is a good one because it’s specific enough and “Talk to bitcoin” or “Talk to cryptocurrency” is not good because it’s describing a category and not a specific function.
- Here we have listed some of the resources, that help you to get started with building your Action as well as support/documentation