Presented at FITC Toronto 2017
More info at http://fitc.ca/event/to17/
Presented by Rami Sayar, Microsoft
Imagine a platform where language is the new UI. When we build chat bots that augment human abilities and experiences, it’s not about man versus machine. It is about man with machines, and each excels at different things. Machines are incredibly persistent if we instruct them to be; even if all we want them to do is provide us with an endless series of cat GIFs. And what better way to please the human spirit than by randomly sending you a cat GIF and being able to answer your questions!
This talk will introduce Microsoft’s Conversation as a Platform and dive into the Bot Framework and Microsoft Cognitive Services. Rami will show you how we built the FITC bot, the design considerations and conversation flow followed by the reasoning behind our insistence on cat GIFs.
Assumed Audience Knowledge
Five Things Audience Members Will Learn
What are chat bots?
Designing a conversation tree for the FITC Bot.
How do you build a conversation tree with the bot framework?
Understanding how to use natural language processing APIs in your bots
Learning about the fabulous Cat API.
Microsoft Xiaoice now
• Natural Language Chat Bot
• Over 20 million registered users
• The average user interacts with the
service 60 times a month
• 850 thousand followers on Weibo
• Ranked as Weibo’s top influencer
Messaging and Mobile
Platforms • Mobile is the new scale.
• Every app can access your address
book, getting an instant social
graph. The phone number in
particular acts as a unique social
• They can access the photo library
and camera directly (and location),
making sharing easy.
• Push notifications mean you don’t
need people to keep checking your
site (or open emails).
Once in a Decade Opportunity
• Users spending more time on messaging apps
• Opportunity for more natural experiences
• High demand from companies of all sizes
• Simpler deployment/update model than apps
• App ecosystem overly crowded
• Everybody wants to automate something
Is It Here to Stay?
• Messaging & Social are the top experiences across all platforms.
• All the large players are bought in.
• Language understanding has progressed considerably.
• Conversational UI is familiar.
• AI & Machine Learning are now mainstream.
• Cognitive services are accessible.
Conversational UI +
are the differentiators. Don’t
be a Clippy.
Building a Bot for FITC
And also finding a way to troll Shawn
An X-ray of a Typical Bot
Your bot code
+ Microsoft Cognitive Services
Message Input <> Output
Conversation Canvas /
Other services, APIs,
Azure Search, etc…
Bot Builder SDK
Bot Builder SDK
Your code goes here
Your conversation logic
Your Bot Code
• Build with C# or Node.js
• You host your bot wherever
• Dialogs to model a
Bot Framework – Connector
Service Interact naturally wherever your
users are talking
• Dialogs to Model a Conversation
• Dialogs are reusable
• Conversations are scalable to multiple
• Dialog state is persisted in Bot Connector
(per-user, per-session, per- user+session)
• Types of Dialogs:
• Built-in prompts
• Yes/No, String, Number, Choices
• Natural language understanding (LUIS)
• Form slot filling (branching, disambiguation,
• Profile (e.g., home address)
Bot Builder – Attachments
• Attachments property
• Array of attachment objects
• Send / receive images
• Rich Cards
Name Description Example
The contentType of the
A link to content of type
An embedded object of type
If contentType =
"application/vnd.microsoft.hero" then Content
would be a JSON object for the HeroCard
Bot Builder – Rich Cards
• Hero and Thumbnail cards
• multipurpose card;
• a single large or small image
• One or more button
• and a "tap action“
• along with text content to display on the card.
• Receipt card
• a single small image
• a button
• and a "tap action“
• text content
• Carousel card
• Multiple cards
• Navigation between cards
• Each has ImageButtonsText
Getting your Bot Project
Creating your Bot
• Create the connector
• We'll start with the console connector
• Create the bot
• Pass in the connector
• Add logic
• Done through dialog
• Communicate with the user
• Done through the session
Diving into the FITC Bot – Find
• Register the bot
• Configure desired channels
• Publish in the Bot Directory.
• All bots registered with Bot
Framework are auto-configured
to work with Skype and the Web.
• Public Directory of Bot
• Users can discover, try, and add bots to
the conversation experiences on which
the bot is configured (no messenging
• Bots are public at developer discretion;
bots must be submitted for review and
approved in order to appear in the
• Lets you understand what your
users are saying (e.g. “Read me
the headlines” or “Pause for 5
• Determines intent
• Detects entities
• Seamless integration with Speech
• LUIS learns over time
• Use pre-built, world class models
from Bing and Cortana
• Models work across Internet
Intelligent Service Intent: TextToSpeech
Read me the
DateTime.duration: 5 Minutes
Content: Daily Show
DateTime.date: T-1 Day
• Analyze an Image – Understand
content and features within an
image for a growing set of
• Wide range of concepts from
cat to fireworks
• OCR – Detect and recognize words
within photos or scanned
• Generate Thumbnail – Scale and
crop an image, while retaining key
• Powers Bing Image search and
A GOAL WITHOUT
A PLAN IS
JUST A WISH
Category People; 1 face found
Adult / Racy? False / True
Black & White? No
• Face Detection – Detects faces
and their attributes within an
• Face Verification – Check if two
faces belong to the same person
• Similar Face Searching – Find
similar faces within a set of
• Face Grouping – Organize many
faces into groups
• Face Identification – Search which
person a face belongs to
Face APIs Face Rectangle – width; height; left; top
• Age – 31
• Gender – Male
Face Landmarks –pupil, nose, mouth
He is Chao Wang…
• Identifies emotions based on
expressions that psychologic
research identifies as universal:
Happiness, Surprise, Sadness,
Anger, Fear, and Disgust
• Has been trained to identify
• Get insights into how your users
react to content and experiences
• Track facial expressions for
multiple users over time.
Diving into the FITC Bot – QnA
What Makes a Bot Great?
• It is not how “smart” they are
• It is not how much natural language they offer
• It is not whether they use voice or not
• It’s whether it solves the user’s needs in the quickest/easiest way
compared to any other option
• It’s all about user experience... Just like an app, or a website
How Does It React in Crisis?
• Does it provide specific responses that
• Does it confirm actions?
• Is it accessible & inclusive?
• Does it allow human interruption?
• Design for humans, not yourself.
• Did you map out all the paths? You’re
going to need a bigger whiteboard.
The "stubborn bot" The "clueless bot"
The "mysterious bot" The "bot that can't forget"
Satya’s Artificial Intelligence Bots
• Bots must be designed to assist humanity.
• Bots must be transparent.
• Bots must maximize efficiencies without destroying the dignity of people.
• Bots must be designed for intelligent privacy.
• Bots must have algorithmic accountability so that humans can undo
• Bots must guard against bias, ensuring proper, and representative research
so that the wrong heuristics cannot be used to discriminate.