Saloni jaiswal mba-2nd semester_miniproject

1
Mini Project Report
Dr. A.P.J Abdul Kalam Technical University
Lucknow, Uttar Pradesh – 226031
CHATBOT
Submitted by
Saloni Jaiswal
Roll No: 200037070002
Session 2020-21
Under the guidelines of
Prof Dr. Abhishek Sharma
Department of Management
INSTITUTE OF MANAGEMENT EDUCATION
178 G.T SAHIBABAD, GHAZIABAD (INDIA)

2
CERTIFICATE BY GUIDE
Certified that the Mini Project Report entitled Chatbot submitted by Saloni Jaiswal is his/her own
work and has been done under my supervision. It is recommended that this Project be placed before
the examiner for evaluation.
(Signature of the Guide)
Dr. AbhishekSharma
IME College, Sahibabad
College code -37

3
DECLARATION
I the undersigned solemnly declare that the report of the mini project work entitled Chatbot, is
based my own work carried out during the course of MBA II Semester my study under the
supervision of Dr. Abhishek Sharma. I assert that the statements made and conclusions drawn
are an outcome of the project work. I further declare that to the best of my knowledge and belief
that the project report does not contain any part of any work which has been submitted for the
award of any other degree/diploma/certificate in this University or any other University.
Saloni Jaiswal
Roll No: 2000370700031
MBA II Semester

4
ACKNOWLEDGEMENT
I would like to express my special thanks of gratitude to my guide Dr Abhishek Sharma who gave
me the golden opportunity to do this wonderful mini project on the topic Chatbot, which also helped
me in doing a lot of work and I came to know about so many new things I am really thankful to them.
Secondly I would also like to thank my parents and friends who helped me a lot in finalizing this
mini project within the limited time frame.
Saloni Jaiswal

5
TABLE OF CONTENTS
Contents Page no.
 Introduction …………..........................................................................................6
 Company Profile.....................................................................................................14
 Objectives of the Study……………………………………………………………16
 Literature Review...................................................................................................23
 Research Methodology….......................................................................................27
 Data Analysis and Interpretation............................................................................36
 Conclusions…………………………………........................................................48
 Limitations and future work………………………………………………57
 References...............................................................................................................60
 Questionnaire...........................................................................................................61

7
1.1 ABSTRACT
A chatbot is a computer program that can converse with humans using artificial intelligence in
messaging platforms. The goal of the project is to add a chatbot feature and API for Yioop.
discussion groups, blogs, wikis etc. Yioop provides all the basic features of web search portal. It
has its own account management system with the ability to set up groups that have discussions
boards. Groups are collections of users that have access to a group feed. The user who creates a
group is set as the initial group owner. Posts are grouped by thread in a group containing the most
recent activity at the top. The chatbot API for Yioop will allow developers to create new chatbots,
powered by rules or artificial intelligence, that can interact like a human with users in a groups
feed page. Example chatbots that can be developed with this API is weather chatbots or book
flight chatbots. Over past few years, messaging applications have become more popular than
Social networking sites. People are using messaging applications these days such as Facebook
Messenger, Skype, Viber, Telegram, Slack etc. This is making other businesses available on
messaging platforms leads to proactive interaction with users about their products. To interact on
such messaging platforms with many users, the businesses can write a computer program that can
converse like a human which is called a chatbot.
Chatbots come in two kinds:
 Limited set of rules
 Machine learning
Chatbot that uses limited set of rules
This kind of bots are very limited to set of texts or commands. They have ability to respond only
to those texts or commands. If user asks something different or other than the set of texts or
commands which are defined to the bot,
it would not respond as desired since it does not understand or it has not trained what user asked.
These bots are not very smart when compared to other kind of bots.
1.2 Chatbot and Machine learning
Machine learning chatbots works using artificial intelligence. User need not to be more specific
while talking with a bot because it can understand the natural language, not only commands. This
kind of bots get continuously better or smarter as it learns from past conversations it had with
people.

8
Here is a simple example which illustrate how they work. The following is a conversation between
a human and a chatbot: Human: “I need a flight from San Jose to New York.” Bot: “Sure! When
would you like to travel?” Human: “From Dec 20, 2016 to Jan 28, 2017.” Bot: “Great! Looking
for flights.”
In order to achieve the ultimate goal, I have taken an iterative approach and divided my work into
four major deliverables. These deliverables not only helped me in understanding the code structure
of Yioop but also enhances Yioop’s functionality. In the rest of the report, I will be discussing
about the four deliverables. To understand more on chatbot service, I had implemented a Facebook
Messenger Weather Bot in deliverable 1, which is discussed in next section. The purpose of
deliverable 2 is to introduce chatbots to the Yioop. I have added Bot Configuration settings which
is used to add bot users in Yioop. In the next deliverable, I have added a functionality where the
user will be able to call bots in a group thread. Activation of bots will happen by calling respective
callback URL which is already configured that helps bots to have a conversation with users. More
details on this is discussed in deliverable 3 section. As a deliverable 4, I have created a weather
bot i.e, a web application in php that calls yahoo API to get weather information.
I have implemented a Facebook Messenger Bot to get an overview of how chatbot is build. During
this implementation, I understood the flow of control for a chatbot service with other services
which is explained below.
In order to create a Facebook Messenger Bot, a developer needs to be authenticated and approved
by Facebook to converse with the public and the web server for security reasons. For a Facebook
Messenger Bot, I have created a simple web application using Node.js by installing the necessary
dependencies using npm. I ran this locally. I also downloaded and installed ngrok and started it -
npm run ngrok. This launched a Forwarding URL to the local running server that means any
requests to Forwarding URL will hit the locally running server. This url is used as callback URL
in Facebook App which will be explained further. To set up the Facebook App, I have created a
Facebook Page and Facebook App using my Facebook account. While setting up a Webhook in
the app settings, I have given the Forwarding URL as Callback URL and added code for
verification.The access token in page settings is stored as environment variable as it will be used
in integration. In order to make webhook to receive messages from this page, the app is subscribed
to the page created. To set up the bot to handle the POST calls at webhook, I have created a
webhook endpoint in the sample application.

9
1.3 Artificial Intelligence
AI was coined by John McCarthy, an American computer scientist, in 1956 at The Dartmouth
Conference where the discipline was born. Today, it is an umbrella term that encompasses
everything from robotic process automation to actual robotics. It has gained prominence recently
due, in part, to big data, or the increase in speed, size and variety of data businesses are now
collecting. AI can perform tasks such as identifying patterns in the data more efficiently than
humans, enabling businesses to gain more insight out of their data.
Types of artificial intelligence
AI can be categorized in any number of ways, but here are two examples.
 The first classifies AI systems as either weak AI or strong AI. Weak AI, also known as
narrow AI, is an AI system that is designed and trained for a particular task. Virtual
personal assistants, such as Apple's Siri, are a form of weak AI.
Strong AI, also known as artificial general intelligence, is an AI system with generalized
human cognitive abilities so that when presented with an unfamiliar task, it has enough
intelligence to find a solution. The Turing Test, developed by mathematician Alan Turing
in 1950, is a method used to determine if a computer can actually think like a human,
although the method is controversial.
 The second example is from Arend Hintze, an assistant professor of integrative biology
and computer science and engineering at Michigan State University. He categorizes AI into
four types, from the kind of AI systems that exist today to sentient systems, which do not
yet exist. His categories are as follows:

10
Type 1: Reactive machines.
An example is Deep Blue, the IBM chess program that beat Garry Kasparov in the 1990s. Deep Blue can
identify pieces on the chess board and make predictions, but it has no memory and cannot use past
experiences to inform future ones. It analyzes possible moves -- its own and its opponent -- and chooses the
most strategic move. Deep Blue and Google's AlphaGO were designed for narrow purposes and cannot
easily be applied to another situation.
Type 2: Limited memory.
These AI systems can use past experiences to inform future decisions. Some of the decision- making
functions in autonomous vehicles have been designed this way. Observations used to inform actions
happening in the not-so-distant future, such as a car that has changed lanes. These observations are not
stored permanently.
Type 3: Theory of mind.
This is a psychology term. It refers to the understanding that others have their own beliefs, desires and
intentions that impact the decisions they make. This kind of AI does not yet exist.
Type 4: Self-awareness. In this category,
AI systems have a sense of self, have consciousness. Machines with self-awareness understand their current
state and can use the information to infer what others are feeling. This type of AI does not yet exist.

11
Examples of AI technology
Automation is the process of making a system or process function automatically. Robotic process
automation, for example, can be programmed to perform high-volume, repeatable tasks normally performed
by humans. RPA is different from IT automation in that it can adapt to changing circumstances.
Machine learning is the science of getting a computer to act without programming. Deep learning is a
subset of machine learning that, in very simple terms, can be thought of as the automation of predictive
analytics. There are three types of machine learning algorithms: supervised learning, in which data sets are
labeled so that patterns can be detected and used to label new data sets; unsupervised learning, in which
data sets aren't labeled and are sorted according to similarities or differences; and reinforcement learning,
in which data sets aren't labeled but, after performing an action or several actions, the AI system is given
feedback.
Machine vision is the science of making computers see. Machine vision captures and analyzes visual
information using a camera, analog-to-digital conversion and digital signal processing. It is often compared
to human eyesight, but machine vision isn't bound by biology and can be programmed to see through walls,
for example. It is used in a range of applications from signature identification to medical image analysis.
Computer vision, which is focused on machine-based image processing, is often conflated with machine
vision.
Natural language processing (NLP) is the processing of human -- and not computer -- language by a
computer program. One of the older and best known examples of NLP is spam detection, which looks at
the subject line and the text of an email and decides if it's junk. Current approaches to NLP are based on
machine learning. NLP tasks include text translation, sentiment analysis and speech recognition.
Pattern recognition is a branch of machine learning that focuses on identifying patterns in data.
Robotics is a field of engineering focused on the design and manufacturing of robots. Robots are often used

12
to perform tasks that are difficult for humans to perform or perform consistently. They are used in assembly
lines for car production or by NASA to move large objects in space. More recently, researchers are using
machine learning to build robots that can interact in social settings.
1.4 AI applications
AI in healthcare. The biggest bets are on improving patient outcomes and reducing costs. Companies are
applying machine learning to make better and faster diagnoses than humans. One of the best known
healthcare technologies is IBM Watson. It understands natural language and is capable of responding to
questions asked of it. The system mines patient data and other available data sources to form a hypothesis,
which it then presents with a confidence scoring schema. Other AI applications include chatbots, a computer
program used online to answer questions and assist customers, to help schedule follow-up appointments or
aiding patients through the billing process, and virtual health assistants that provide basic medical feedback.
AI in business. Robotic process automation is being applied to highly repetitive tasks normally performed
by humans. Machine learning algorithms are being integrated into analytics and CRM platforms to uncover
information on how to better serve customers. Chatbots have been incorporated into websites to provide
immediate service to customers. Automation of job positions has also become a talking point among
academics and IT consultancies such as Gartner and Forrester.
AI in education. AI can automate grading, giving educators more time. AI can assess students and adapt to
their needs, helping them work at their own pace. AI tutors can provide additional support to students,
ensuring they stay on track. AI could change where and how students learn, perhaps even replacing some
teachers.
AI in finance. AI applied to personal finance applications, such as Mint or Turbo Tax, is upending financial
institutions. Applications such as these could collect personal data and provide financial advice. Other

13
programs, IBM Watson being one, have been applied to the process of buying a home. Today, software
performs much of the trading on Wall Street.
AI in law. The discovery process, sifting through of documents, in law is often overwhelming for humans.
Automating this process is a better use of time and a more efficient process. Startups are also building
question-and-answer computer assistants that can sift programmed-to-answer questions by examining the
taxonomy and ontology associated with a database.
AI in manufacturing. This is an area that has been at the forefront of incorporating robots into the workflow.
Industrial robots used to perform single tasks and were separated from human workers, but as the
technology advanced that changed.

15
Chatbots has emerged as a hot topic in the latest years, and it is used by numerous
companies in various areas - help desk tools, automatic telephone answering systems,
e-commerce and so on. Even though the technology has been around since the 60’s (Atwell &
Shawar, 2007). Why are we suddenly so interested in this technology now? This can likely be
explained by the recent year's advancements in messaging applications and AI technology
(Brandtzaeg & Følstad, 2017).
In the article Chatbots: Are they really useful? Atwell and Shawar provide real-life examples of
different chatbots in different contexts. One of the examples is Sophia, a robot that was
developed to assist in mathematics at Harvard by answering students questions. This turned
out to be applicable in many other contexts. Living in Norway you have probably noticed
“Kommune Kari”. A chatbot that many of the municipality have available on their web-pages.
Kari is there to answer “easy” questions like “when will the garbage truck come?” and “where
can I find available jobs?”. Kari’s goal and the job is to provide information so that you as a user
don’t have to navigate the “massive information flow” (Schibevaag, 2017). This way of using a
chatbot is a part of the Question Answering (QA) field which is a combination between AI and
information retrieval (Molla & Vicedo, 2007). QA can be defined as:
“... the task whereby an automated machine (such asa computer)answers arbitrary questionsformulated in
natural language.QA systems are especially useful in situationsin which a user needs to knowa very
specificpiece ofinformation anddoesnothave thetime—orjustdoesnotwant—to read all the available
documentation related to the search topic in order to solve the problem at hand”. (Molla & Vicedo, 2007).
Sophia and Kari are examples of chatbots that operate in “very specific” domains. This means
that if you were to ask Kari about math and Sophia about when the garbage truck comes none
of them would know the answer - because the question is outside of their domain. Chatbots
have what is called a natural language user interface and therefore
communicate with users via natural language ㅡ how a human would talk on a regular basis
(Brandtzaeg & Følstad, 2017). Therefore they use what is called natural language processing
(NLP) where the chatbot uses computational techniques to analyze text, where the goal is to
produce a human-like answer based on a linguistic analysis (Hirschberg & Manning, 2015).
For a chatbot to be especially useful to a certain domain some criteria have to be met. Minock (2005)
proposes the following criteria for a domain to be successful in answering domain-specific questions: a
domain should be circumscribed, complex and practical. This is summarized in the table below

17
2.1 Objectives
The last few years have seen a rapid surge in on-demand messaging that has shifted consumers’ way
of communicating with brands. To provide superior customer service, more and more businesses
today are integrating chatbots into their processes.
In specific industries where high-volume customer interaction is at the center of the business, such
as banking, insurance, and healthcare, chatbots have been complete game-changers. They help save
over 4 minutes on average per customer inquiry, compared to the executives answering the calls, with
a high success rate per interaction.
In this article, we will explore the key benefits of chatbots for both businesses and customers, along
with the factors to take into consideration while building powerful chatbots.
2.1 Benefits Of Chatbot For Businesses
There are numerous benefits to using chatbots, and it largely depends on how businesses and
stakeholders can leverage them to enhance the customer’s experience.
Here are some of the top benefits of using a chatbot to improve your business efficiency:

18
1. Cost Savings
With a fiercely competitive business landscape today, businesses’ need for a robust customer service
department is consistently rising. Implementing powerful chatbots allows companies to manage a
massive amount of customer queries in relatively short periods.
Although chatbot implementation requires a certain amount of investment, this is significantly lower
than the traditional customer service model that includes infrastructure, salaries, training, and
multiple other resources.
Research also suggests that businesses every year spend nearly $1.3 trillion to service almost 265
billion customer requests, and chatbots can help businesses save up to 30%! Chatbots help businesses
optimize their costs without compromising their customer service quality. Chatbots can
 Automate day to day business processes and allow the customer support team to
concentrate on more complex queries
 Systematically scale their chat support during peak hours to deliver quality support
and enhance customer satisfaction
 Enable multiple new customer service models to help increase brand face value and
credibility
2. Offer Website Visitors Contextual, AI-Driven Support
Contrary to the popular belief that a chatbot’s main benefit is just answering queries and offering
customer support, chatbots can provide value-driven, contextual support that can assist businesses
significantly.
An AI chatbot uses the data to provide a personalized experience to the users. These chatbots go
much beyond just answering pre-programmed questions that every customer will experience in a
precisely similar way.

19
3. Better Analysis of Customer Data
With the help of chatbot analytics, businesses can analyze how well the bot performs in terms of
successful business outcomes and sales generated and detailed insights on how people engage with
the business and what they are asking for.
Apart from this, chatbots are flexible in their approach and allow businesses to serve their clients on
almost every platform. It’s quite simple and easy to adopt a chatbot to various platforms and integrate
them into your existing IT infrastructure.
4. Enhances Customer Engagement And Sales
Customer engagement is the critical requirement to boost your sales and keep your customers
engaged, and chatbots are an excellent tool for this. Research suggests that businesses that
successfully engage with their customers can increase the customer spend by almost 20% to 40%!
These chatbots’ flexible structure makes them super easy to integrate with other systems, increasing
customer engagement in return. An excellent example of this would be getting reservations online.
As soon as the customer starts communicating with the chatbot and shows interest in booking, the
chatbot immediately leads them to the booking page in an attempt to close the sale.
This kind of quick and hassle-free experience leaves the customer happy and satisfied. Further, due
to chatbots’ programmed nature, they sound more natural and human-like, making the customer’s
experience more positive and pleasant.
5. Better Lead Generation, Qualification & Nurturing
A chatbot is equipped to ask necessary and relevant questions, persuading the customers, and
generating leads quickly. It ensures that the conversation flow is in the right direction to get higher
conversion rates.

20
Apart from generating leads, another benefit of chatbot is that chatbots can help you qualify leads
through identified KPIs, including timeline, budget, relevancy, resources, and more, to prevent you
from dealing with time-consuming leads.
6. Bots Save A Great Deal Of Time
One of the benefits of chatbots is that chatbots empower businesses and save time by solving basic
queries. Only the complex queries that need human input are directed to the executives on the support
team.
Chatbots do this by quickly resolving customers’ questions and automating information-based
queries so that support staff can spend more time on crucial issues that need human support, reducing
operational costs, time and manpower significantly.
7 . Massive Range Of Possible Applications
One of the distinct advantages of chatbots for businesses is that they offer a wide range of applications
and are not limited to the single-use case of answering customer questions.
Some of these everyday use cases of chatbots include –
 Marketing: Chatbots can be used for multiple marketing activities, including lead
generation, data collection, increased custom interaction, and product consulting.
 Sales: Helps in the qualification of leads and supports throughout the sales funnel.
 Customer Service: Assists in answering FAQs and offers dedicated support in case
of problems.
 IT Service Helpdesk: Offers support for internal or external service desk applications.
8. Applicable To Multiple Industries
Regardless of the industry, chatbots today are beneficial to every type of business and industry out
there.

21
In specific, there are a few industries that are more likely to be revolutionized from AI-based chatbots.
Some of these are –
 Healthcare
There are multiple benefits of chatbots in the healthcare industry, including booking
appointments, refilling prescriptions, and sending medical details. Additionally, these
chatbots can also provide medical assistance to patients to monitor their health periodically
and remind patients to take medicines.
 Banking & Financial Sector
Chatbots offer an excellent way to revolutionize the heavily transactional activities of banks
and financial institutions. One of the benefits of chatbots in banking is answering customer
questions about online banking and giving them information about account opening, card
loss, and branches in various locations.
 Education
There are several benefits of chatbots in education, such as intelligent tutoring systems and
a personalized learning environment for students. Additionally, chatbots can also analyze a
student’s response and how well they learn new material or assist in teaching students by
sending them lecture material in the form of messages in a chat.
 HR
Implementing chatbots in HR and recruiting can help in multiple ways by automating each
recruiting process stage. Right from searching for candidates, evaluating their skills, and
informing them if they are qualified for a particular job posting, the uses of chatbots are
many.

22
 Retail
Another important industry for chatbot application is retail and e-commerce. For instance,
businesses can use retail chatbots to answer customer questions while they shop online,
offering more personalized product recommendations, streamlining the sales process or
helping customers search for a product, place an order, make payment for it, and track the
delivery.
 Travel & Tourism
Chatbots are quite popular in the travel and tourism industry. Chatbots in the travel
industry can answer questions about bookings by offering their visitors information on how
to get there or the current weather conditions.

24
3. Literature
Tab 2.1 Literature Survey
3.1 Natural Language Processing
Natural Language Processing (NLP) is the study of letting computers understand human
languages. Without NLP, human language sentences are just a series of meaningless symbols
to computers. Computers don’t recognize the words and don’t understand the grammars. NLP
can be regard as a “translator”, who will translate human languages to computer
understandable information.
Traditionally, users need to follow well-defined procedures accurately, in order to interact with
computers. For example, in Linux systems, all commands must be precise. A single replace of

25
one character or even a space can have significant difference. However, the emergence of NLP is
changing the way of interacting. Apple Siri and Microsoft Cortana have made it possible to give
command in everyday languages and is changing the way of interacting.
3.2 Machine Learning.
Machine Learning (ML) is an area of computer science that "gives computers the ability to learn
without being explicitly programmed". The parameter of the formulas is calculated from the data,
rather than defined by the programmer. Two most common usage of ML is Classification and
Regression. As shown in figure1, Classification means to categorize different types of data, while
Regression means to find a way to describe the data. Basic ML program will have two stages,
fittingand predicting.In the fitting stage, the program will be given a large set (at least thousands)
of data. The program will try to adjust its parameter based on some statistical models, in order to
make it “fit” the input data best. In the predicting stage, the program will give a prediction for a
new input based on the parameters it just calculated out. For example, the famous Iris flower
dataset [9] contains the measurement of several features of three different species of flowers, such
as the length of sepals and petals. A well-defined ML program can learn the pattern behind this
feature and give prediction accordingly.

26
Figure2a: Classification Figure2b: Regressing

28
4. Research Methodology
Artificial Intelligence is the driving force behind the creation of innovat i ve
products like autonomous vehicles and chatbots. Recent advancements in
Natural Language Processing (NLP) have made chatbots, also referred to as
virtual assistants, a great option for improving the customer experience.
Answering frequently asked questions, filing claims, checking the status of
an order and getting feedback from customers are among the most popular use
cases for chatbots. Building a chatbot that offers a good experience to
customers requires collaboration from an interdisciplinary team of business
analysts, service designers, data scientists, machine learning engineers and
software developers. The chatbot development methodology blends several
modern frameworks and methodologies including design thinking, AI
innovation sprints, and agile software development.
Method
To gain insight into chatbot user experience, we saw it as necessary to gather
data from a sample of participants experienced in chatbot use. Furthermo re,
because we aimed to explore characteristics of chatbot user experience across
chatbots and application domains, we needed a sample that included
participants who were experienced with different chatbots.
For this purpose, we applied a questionnaire study. Here, we aimed to involve
a relatively large sample of participants with diverse backgrounds in chatbot
use and have them report on their chatbot user experience.
Participant recruitment
Chatbots are not yet in common use for the majority of information and
communication technology users. Hence, participant recruitment and filter i ng
were important parts of the study method. We decided to recruit a US sample
because important platforms for chatbot use, such as Facebook Messenger,

29
Skype, and Kik, target this geographical market in particular. We also decided
to involve only relatively young participants (16–55 years of age), as we saw
these as more likely to be chatbot adopters given their more frequent use of
messaging platforms.
Data were collected in two waves: April and June 2017. The recruitment of
study participants and administration of data collection were done by Survata,
an independent research company. Survata samples participants through
partnerships with online publishers. Participants gain access to premium
content as an incentive for participation. In our study, participants were
invited to complete the questionnaire following an introductory filter i ng
question about their experience with chatbots. Findings based on the first
wave data collection have previously been presented by Brandtzaeg and
Følstad.
Study material
The first of the two items of particular interest for this study concerned
positive episodes of chatbot use. The participants were asked to ‘tell about
one really good experience that you have had with a chatbot’. They were
specifically requested to use their own words and to be as detailed as possible.
The second of these two items concerned negative episodes. Here, the
question was posed in two different ways. For the first wave of data collect io n
(April 2017), the participants were asked to ‘tell about a really bad experience
you have had with chatbots’ and were asked to use their own words and
provide as much detail as possible. For the second wave (June 2017), the
participants were asked, ‘Have you ever stopped using a chatbot? Please
explain the most important reason to stop using it.’ The reason for modifyi ng
how this question was asked across the two waves of data collection was that
a surprisingly large proportion of participants in the first wave did not report
negative episodes of use. We therefore decided to change the question to what
we saw as a relatively stronger negative wording. The findings for these two

30
versions of the negatively worded questionnaire item, however, did not differ
substantially and were therefore analysed as one data set.
The design of the two qualitative questionnaire items was motivated by the
critical incident technique (CIT), a qualitative approach to data collect io n
with roots in the field of psychology. Using CIT, researchers gather users’
stories about salient incidents during their exposure to a particular
phenomenon in order to identify and understand the factors that are import ant
in driving their experience. This technique is particularly beneficial because
it provides a way to gather rich insight into those incidents or characterist ics
that are most important in determining user experience from the perspective
of the user. As such, CIT is a potentially valuable technique for investigat i ng
chatbot user experience. We are not aware of other studies that have applied
a CIT approach for this purpose.
Data analysis
Prior to analysis, the dataset was filtered to ensure that all participants had
sufficient experience with chatbots and that there was sufficient variation in
the chatbots used by the participants. This filtering was based on the
participants’ responses to the question, ‘What is the name of the latest chatbot
you used?’. We saw this question as a good indicator that (a) the participants
actually had knowledge of chatbots, and (b) the participants were not
confusing the term chatbot with chat platforms for human–human interact io n.
All participants who reported a name corresponding to a known chatbot or
who were able to describe such a chatbot if they didn’t remember the specific
name were included for analysis. The responses of the remaining participants
were filtered out.
The participant reports were analysed through thematic analysis [14], where
a set of codes was identified on the basis of the themes emerging in the
qualitative data. This set of codes was then grouped according to

31
Hassenzahl’s [23] framework of pragmatic and hedonic attributes and used to
analyse all participant reports. Note that some of the codes represent
attributes not anticipated in the pragmatic- hedonic framework, for example
attributes that reflect the social and humanlike characteristics of chatbot
interaction.
Each participant report could be coded as corresponding to one, several or
none of the codes. Participant reports that did not correspond to any of the
codes were coded as ‘other’.
Coding participant responses to reflect pragmatic or hedonic attributes also
allowed us to investigate the effect of age on the likelihood that one or the
other of these two attribute types would be emphasized in participant
responses. To determine this, we conducted two bivariate correlatio n
analyses: one for participants’ tendency to report on pragmatic attributes and
one for their tendency to report on hedonic attributes. These two analyses
were conducted using the statistical software package IBM SPSS Statistics
25.
Defining the Chatbot’s Purpose and Managing Expectations
Like most artificial intelligence applications, the first step in developing a
chatbot is clearly defining its purpose, the problem it is going to solve, and
the value it is going to bring to the users. Chatbots should be designed to help
users navigate through a very specific business scenario. One must avoid the
trap of trying to create a jack of all trades (and master of none) chatbot. For
example, some conversational applications are getting a lot of hype that can set
unrealistic expectations and contribute to negative user experiences. Defining a narrow
scope and a clear chatbot purpose will be key for managing the rest of the
chatbot development process.
Understanding the Audience

32
As expected in any human-centric design approach, it’s important to
understand the target audience. Service designers will conduct interviews to
gain an understanding of customers as human beings. Using design think ing
techniques, we learn about their pain points, their likes and dislikes, how they
communicate and what they value the most. Understanding the human needs
of a customer is essential to designing a chatbot experience that delights
them.
A Chatbot Design Experience is a great tool to have to guide the team in defining
all the key components of a virtual assistant.
Defining the Chatbot’s Personality
Techniques to define user personas should be adapted to create a “chatbot
persona”, including its age group, interests, how it acts, how it communicat es,
its sense of humor and its limitations. Is the chatbot an advisor, an assistant,
or a friend? A poorly designed chatbot personality is one of the top reasons why
many chatbots fail. A chatbot with personality and empathy toward the
customers can drive user engagement and create meaningful experiences.
Robotic chatbots, however, have the completely opposite effect.
Designing the User Journey and Conversation
With user persona and chatbot personalities created, a service designer next
should define in greater detail the human + machine interaction. He/she must
design the user journey and script the actual conversations. What quest ions
might a user answer? If the chatbot does not know an answer, do they redirect
the query to a support representative? What information does the chatbot need
to know in order to answer a question? The conversational and human-in-t he-

33
loop design are the most challenging parts of designing an optimal customer
experience.
Developing the Chatbot
Once the chatbot design is complete, the development team can map out the
features and create a technical design. Identifying if the chatbot requires
artificial intelligence (AI) to fulfill its purpose will be important in defining
the chatbot development methodology. Not all chatbots need AI to be fully
functional, and if this is the case, the team will likely implement agile
development methodologies in the traditional software development lifec yc le
(SDLC). Beyond building a rule-based chatbot, the software engineering team
should define what background tasks and fallback mechanisms need to be
carried out in order to properly integrate all the system’s components .
Development can be completed through incremental deliveries, adapting
design with feedback obtained from the end-users.
If a chatbot requires Natural Language Processing (NLP) components, agile
software development and machine learning will need to be blended in a
chatbot development methodology. The machine learning process is
inherently different from rule-based systems. It requires exploration and
experimentation when performing data collection, data preparation (cleaning,
wrangling, and merging), feature engineering, model training, and model
evaluation. The project team must account for the experimentation that will
take part in the chatbot development process. Wovenware has developed a
proprietary methodology, the Innovation Sprint, to integrate innovation and
discipline in a goal-oriented experimentation process.
Integrating Natural Language Processing
The NLP components in chatbots are mainly used to recognize user intent and
extract entities. Each intent can represent a task to be performed. A
conversation flow will vary depending on the detected intent. How the chatbot

34
handles intents will define its relationship with the user. Optional
components, like sentiment analyzers and language translators, can add value
and enhance chatbot experience. Sentiment analysis can help chatbots respond
to users in a personalized manner by understanding what makes them happy,
while language translation components can be used to give chatbots
multilingual capabilities. Data scientists will optimize NLP components in
order to provide information about intent, meaning and the context in which
the chatbot needs to appropriately respond to a user’s query. Natural
Language Processing is constantly evolving so these components must be
updated regularly to keep the virtual assistant functioning in optimal
conditions.
Testing
Once each system component has been designed and implemented, testing and
evaluation should be carried out. For middleware software, unit testing should
be performed to test individual components. Machine learning components
need to be evaluated differently. Testing data is used to evaluate the trained
models and success metrics must be clearly defined in order to measure model
performance. Usability testing must be done before and after deployment.
These tests will help to understand if the chatbot follows the expected
conversational flows and error handling strategies by monitoring user
interactions.
From Chatbot Rookie to MVP
Successful chatbots are designed to learn, making maintenance an integra l
part of the chatbot development methodology. Once the chatbot is interact ing
with real users it is important to analyze user feedback and sentiment, along
with other insights the interactions may produce. Insight analysis may give
us an understanding of possible usability issues or areas of improvement, but
it also may provide us with possible market opportunities to implement more
data-driven solutions. Analytics can help us understand bot performance, user

35
engagement, sentiment and demographics. Fixing bugs and other software
defects should be part of the software maintenance strategy, but with evolving
customer expectations and technology advancements it is important to also
keep the system updated and sustainable. Metrics for the machine learning
components should always be monitored in order to address a possible
decrease in performance. In the case of chatbots, the ever-evolving nature of
natural language may cause the NLP components to worsen over time.
Constant retraining, through active learning and evaluation of those
components is vital during the maintenance and development process.
Chatbots move from rookie to MVP level through the collaboration of
business leaders, service designers, data scientists, machine learning
engineers and software developers. Each member of the team needs to stay at
the top of his/her game to implement modern frameworks and novel
technologies to continue to improve the customer experience. The team will
always be on a journey to understand the user, to evolve the applicat io n
around the things that matter to them the most and make the technology as
human as possible, but this is what makes for great chatbots.

36
Data Analysis and Interpretation

37
Data Analysis and Interpretation
Web focused metrics do not calculate misunderstood requests or message
delays. Chatbots require need new metrics and to get the most out of their
benefits. If you are already familiar with web metrics, jump to chatbot
specific performance metrics such as message metrics or bot metrics
We have previously written an in-depth guide on conversatio na l
bots/chatbots. We argued that chatbots will change the world and brands
should start experimenting with them. Chatbot analytics is also referred to as
conversational analytics, chatbot analytics, bot analytics, chatbot
intelligence, which is an important tool in guiding brands’ experiments with
chatbots. We cover all relevant question about bot analytics tools and
important chatbot metrics that a company needs to measure the progress of
its chatbot/conversational AI system:
 What is chatbot analytics?
Chatbot analytics is the process of analyzing historical bot
conversations to gain insights about chatbot performance and
customer experience.
 Why is it important now?
A business’s work as a chatbot developer doesn’t end once their bot
goes live. Due to increasing competition in every industry, customer
experience became the key driver in gaining a competitive edge. After
a business deploys a chatbot, it is time to track how people are using
it.
 Reasons that make conversational analytics important are

38
Better focus: As with other analytics approaches, chatbot analyt ic s
enables organizations to track relevant chatbot KPIs to make data-
driven decisions to enhance the chatbot performance. Many chatbot
projects fail due to various reasons including optimizing the wrong
metrics. Relying on chatbot analytics helps businesses avoid potentia l
failures.
Improved data collection: As new data privacy regulations like GDPR
and CCPA make the process of using third-party consumer data
harder, conversational analytics software becomes a tool to
collect first-party data insights since consumers are willi ngl y
interacting with bots.
Customer insight generation: These platforms enable businesses to
map popular user paths, tasks, and exit points via chatbot analyt ic s
dashboards (in visual context) so that organizations can surface
patterns, trends, and correlations that might not get noticed using text -
based data analysis methods. This helps businesses better understand
the customer journey.
 What is a metric?
A metric is a quantifiable measure that is used to track and assess the
status of a specific business process
Defining metrics is important. Since many of the capabilities of the
chatbot will be measured through those metrics. For a newly created
chatbot, those metrics can fluctuate dramatically. After implement i ng
the chatbot, companies need to monitor it closely. Since the
expectations are efficiency enhancement, faster response, and greater
conversion, companies need to define the right metrics. This way, the
performance of the bot can be monitored and improved efficiency. For
data scientists, this dimension of the issue provides many use cases.

39
(User Metrics)
1- Total Users: This is the most basic metric. It captures the number of
people using your chatbot. This matters because its trend shows the change
in the number of users and therefore the amount of data your chatbot has been
exposed to. Also, this would provide critical information regarding the market
size calculations and potentially the effect of your chatbot.
2- Active Users: Active users can be defined as the people who read a
message in the chatbot in a defined time frame. These are your potentia l
targets. Measuring the potential effects of a promotional activity can be
estimated from that number. The number of people who read your message is
critical. This is something like the promotional content on social media.
Engagement is not guaranteed but, the content is seen by the people
3- Engaged Users: Those users are the ones who communicate with the
chatbot. They receive and send messages. This is important since your chatbot
will be able to provide the conversation statistics based on this sub-sample.
They are likely to shape your decision regarding the effectiveness of the
chatbot. It doesn’t make any sense if the chatbot is not able to start the
conversation with the users.
4- New Users: This metric captures the overall success of your chatbot
promotion campaign. New users will be necessary to keep an active user
number. Customers’ preferences changes over time and the amount of
interaction with the chatbot shows an exponential decay. For that reason, new
users will keep your customer base strong.
5- User sentiment: This metric is captured by performing sentiment analys is
so that you can categorize messages as positive, neutral, or negative. You can
gain insights into the user experience and where/when the conversation went
wrong.

40
Message Metrics
First four metrics capture the overall trend in your user base, but you will be
needing greater detail regarding how an individual interacts with your
chatbot. One such metric group is the message metrics.
6- Conversation Starter Messages: This is the number of messages
where the bot starts the interaction. This is critical for measuring the organic
reach of your platform. It is possible to elicit a response by sending messa ges
to users, but as time moves forward, companies would need a lower number
for that metric. Since we will implement chatbot for customer relatio ns
management and digital marketing, after the initial greeting, we need
continuing users to send messages to chatbot directly.
7- Bot Messages: Bot messages are the total number of messages sent by the
chatbot in each interaction. This measures the length of a conversat io n
between a customer and the chatbot. We normally want the number of
messages to be high, but there is one critical condition, our chatbot needs to
respond correctly. In case of misunderstanding or failure to comprehend the
input by the user, the chatbot will say similar words repeatedly.
8- In Messages: This category shows the messages sent by the user. We need
to see whether the user engages with the chatbot or not. If this category is
significantly low, we don’t need to use a chatbot. Using regular social media
such as a Facebook Page or a Twitter account may make more sense, rather
than using a Facebook Messenger chatbot or a voice-assisted technology.
9- Miss Messages: Miss messages are the ones our chatbot can’t process. This
metric may be hard to calculate. Requires the times the chatbot miss interprets
the input. This would be a key metric if the firm starts to engage in countries
where the language is used more idiomatically.

41
10- Total Conversations: Number of conversations started and successfull y
completed on a given day. This is the concept engaged users
11- New Conversations: Number of new conversations started. This captures
both the inexperienced users and the conversations that are initiated by the
returning users on a different matter, problem, or order.
Startups such as Pulse Chat focuses on measuring the number of messages
and provides the details in their platform. Their innovative solution makes it
possible to maintain and improve the quality of the conversation instantly.
The above-mentioned metrics are the key metrics for our metric construct io n.
We know that the other traditional digital marketing metrics can be
constructed using those basic metrics.
Bot Metrics
We will now introduce some other metrics that are critical for measuring the
performance of a chatbot.
12- Retention Rate: This is the percentage of users that return to using the
chatbot on the given time frame. This important since we need to keep the
customer engaged to extract valuable insight regarding the customers’
preferences by making them spend time on the chatbot. Higher retention rate
can be achieved by promotional campaigns such as chat-to-receive-disco unt
or a lottery like a word guessing games. The important is keeping that high
through an organic process. This mostly can be achieved by providing a high-
quality chatbot that meets the customers’ expectations and needs.
13- Goal Completion Rate (GCR): This captures the percentage of
successful engagement through chatbot. Users will probably try to reach

42
different information or service. For an e-commerce company’s bot relevant
goals can be informing the user about a product’s details or purchasing a
product. This shows that the number of times our bot successfully processed
the input and provided the asked information. There are other ways to utilize
this concept. One way is to mine data through the questions asked by the
users. This would show an overall trend in consumer preferences; hence a
greater emphasis can be placed on that issue while training the chatbot.
In the above graph, we can see an example of such use-case. Users mostly
used the chatbot for scheduling their rides, hence greater emphasis should be
placed on that issue, for keeping the consumer engaged and active.
14/15/16 – Goal Completion time/messages/taps: Chatbots need to provide
a seamless and efficient experience and they have plenty of substitutes such
as company’s web pages or apps. Minimizing the effort to complete a goal
can improve user experience.
17- Fallback Rate (FBR): No robot is perfect; therefore chatbots are
expected to fail sometimes, but what really matters is those failures happen
regularly or in some extreme cases. Fallback Rate captures that informat io n.
This is the percentage of times our chatbot failed or experienced a near-failure
situation. Our aim is to minimize this since we need a chatbot that replaces
humans such as customer service representatives, salespeople, and call center
people. When experiencing a high fallback rate, it might be right to try find ing
new data sources or training sets for improving the performance. Fallback
rate can be examined in three categories :
Confusion Rate: Bot may not understand the question that the user is asking
or vice versa. Confusion triggers are a valuable indicator of how and where a
chatbot needs to be improved. It also provides insights into the quality of the
customer experience.

43
Reset Rate: Sometimes users may want to change the previous response in
the question chain. These cases are also considered as fallbacks. You can add
a reset button to the conversational interface to solve the problem. Tracking
this metric may enable you to identify patterns in customer behavior for
certain questions
Human Takeover Rate: Though sales chatbots are designed to transfer the
user to the human agent so that the lead can turn into sales, there are some
cases where a human agent had to take over due to repeated failure of the bot
or customers prefer to have a comprehensive discussion about the product.
According to a customer experience index survey, 86% of customers believe
that chatbots should always have an option to transfer to a live agent.
18- User Satisfaction: A new metric can be defined through the exit surveys.
Customers or the people engaging with the chatbot can rate their experience
to achieve further product excellence. This can be implemented as a binary
variable such as “did the bot perform well? – Yes, or No” or we can create
more complex evaluation forms to rank and provide points for each differe nt
category. This metric would capture the overall effectiveness of the bot from
the user experience point. directly provided by the user.
19- Virality: Not all bots are viral but bots that motivate the user to include
others in the conversation can achieve viral growth. If your solution can
involve others, that will surely aid growth. You can read an analysis of how
viral features helped the growth of the simple bot, Roll.
20-Self-Service Rate: Frequency of completed conversations without a need
for a second-tier call center. This metric would capture the overall
effectiveness of the bot from your point.
Commercial Metrics

44
21- ROI / payback period: Return on investment (ROI) is critical to know
for any project. Companies need to track, at least on a high level, their
spending for the bot and the benefits it generates. The benefits will depend
on metrics like leads generated, fallback rate, cost per fallback so that
businesses can compare bot’s benefits with other channels.
22-Leads generated: For chatbots in sales, this is one of the most important
metrics. They should lead to a higher number or quality of leads captured.
For chatbots in customer service, this is a less relevant metric however some
customer service bots can also identify users’ pain points and cross/up- sell
opportunities.
These are some of the common key performance metrics for chatbots. For
specific projects, companies sometimes create new, ad hoc metrics to measure
the specific impact of improvement projects if they are not captured by these
metrics. However, for typical use cases, these metrics will cover the most
critical areas.
What are key features of chatbot analytics tools?
Sentiment analysis
Chatbots can capture and process customer sentiments from conversatio ns,
without explicitly asking. Thanks to sentiment analysis, businesses can
understand whether users are responding positively or negatively and make
their bot more user friendly.
Customer segmentation
Chatbot analytics tools can use natural language processing(N LP) to extract
conversational data and combine it with web analytics data such as
demographics to segment customers. With customer segmenta tion, businesses

45
are able to personalize conversations and try out A/B testing to chatbot
landing pages and query responses to see how they can increase engageme nt.
Intent analysis
Some tools like Dashbot provide an intent mapping feature to help developers
check how they match messages to intent categories.
Source: DashBot
Transcript Search
Analytics tools enable businesses to track search full text of transcripts to
track the complete user lifecycle.
Identification of task failures

46
With analytics tools, organizations record and categorize all instances where
a bot fails to complete a task so that developers work on bot’s responses to
improve bot’s performance.
What are example chatbot analytics case studies?
Talkpush
Talkpush is a recruitment platform that leverages conversations and social
media to help companies hire talent. Their recruiting chatbot Stanley can
screen a large volume of candidates for the right fit. Stanley provides init ia l
screening and can pass inquiries to a human, schedule interviews, and engage
candidates while they’re waiting at an onsite interview. Talkpush’s clients
are large-scale enterprises that’s why they wanted to ensure they are
delivering a flawless experience to their customers.
They purchased DashBot analytics tool to improve their chatbot with data-
driven insights. With the conversational analytics tool, Talkpush is able to
identify conversations that need optimization by looking at sentiment points.
Thanks to analytics, they doubled their chatbot response accuracy from 30
percent to 60 percent from January to July 2019.
What are leading chatbot analytics vendors & tools?
1 Botanalytics
2 BotExpress.ai
3 ChatBase
4 ChatBot
5 Chat Metrics
6 Dashbot
7 Enterprise Bot Manager (EBM)
8 Kore.ai

47
Botanalytics is based in San Francisco and is the leading company in
conversational analytics tools. They were established in 2016, their main goal
is to improve human-bot interactions and conversational UI through data. By
offering analytics for bots, they empower bot developers, corporate brands
and agencies to improve their chatbots with historical and real- time
conversation data.

49
Results
About the participant sample
The sample consisted of responses from 207 valid participants. Of these, 62%
were male and 38% were female. Mean age was 27 years (SD = 8). The
participants used chatbots on a variety of messaging platforms, notably
Facebook Messenger (79%), Skype (54%), Kik (38%), Viber (12%), Slack
(10%) and Telegram (4%). Further, 65% reported using chatbots daily or
weekly, 48% reported having used chatbots for 3 or more years and 40% had
experience with Google Assistant.
The chatbots reported as the most recently used by the participants reflected
a wide range of chatbots: for productivity purposes, marketing, customer
service and entertainment. The most reported recently used chatbots were
virtual assistants such as Google Assistant (18%), Siri (7%), Alexa (4%) and
Cortana (4%) and chatbots for social chatter such as Cleverbot (11%), Eviebot
(3%), Mitsuku (1%), SimSimi (1%), Zo (1%) and the no longer operative
Smarterchild (3%). The frequent mention of Google Assistant as a recently
used chatbot may be due to its availability on the Android operating system.
Chatbots are the new Apps! As we have discussed in the above deliverab les,
this project brings the power of chatbots to Yioop and enriches its usabilit y.
Chatbots in Yioop can give a human like touch to some aspects and make it
an enjoying conversation. And they are focused entirely on providing
information and completing tasks for the humans they interact with. The
above mentioned functionality in all the deliverables is implemented and
pushed in to Yioop code. By implementing the above mentioned deliverab les
I was able to add a basic chatbot functionality in to the Yioop. I.e.,
configuring and creating accounts for bot users with bot settings which is
mentioned in deliverable 2, activating a bot whenever a user asks for it via
post in a thread which is discussed in deliverable 3 and as I discussed in
deliverable 4, I have implemented a simple weather chatbot that gives weather

50
information whenever a user ask and Fig. 3 tells that I was also able to
converse with the bot in Yioop. I intend to enhance the system developed so
far in CS298. Next step towards building chatbots involve helping people to
facilitate their work and interact with computers using natural language or
using set of rules. Future Yioop chatbots, backed by machine- lear ni ng
technology, will be able to remember past conversations and learn from them
to answer new ones. The challenge would be conversing with multiple bot
users and multiple user
Future Scope
There are limitations to what has been currently achieved with chatbots. The
limitations of data processing and retrieval are hindering chatbots to reach
their full potential. It is not that we lack the computational processing power
to do so. However, there is a limitation on “How” we do it. One of the biggest
examples is the retail customer market. Retail customers are primaril y
interested in interacting with humans because of nature of their needs. The y
don’t want bots to process their needs and respond accor
Positive chatbot experiences
Participants reported a broad set of positive chatbot experiences. The reports
were on average 105 characters long (SD = 90). Nearly all reports provided
sufficient detail to identify one or more characteristics of the experience.
Although we asked the participants to report on a specific positive episode,
not all of them did. Rather, 45% reported on a specific episode, whereas 22%
reported on their overall experience with a specific chatbot. The remaining
participants made high- level reports of chatbot attributes regarded as positive
without mentioning a specific episode or chatbot. To provide a feel for what
the participant reports look like, we include the following two quotes, which
exemplify reports of a specific episode (P37) and reports of a specific chatbot
(P40).

51
I actually recently interacted with a chat bot about a complaint I had with a
company. The chat bot informed me the correct way and persons to send my
information to. It was quick and easy and I really appreciated this since I was
already quite annoyed (P37)
I get Cortana to tell me a joke—on Windows 10 (P40)
Following Hassenzahl's [23] pragmatic-hedonic framework for user
experience, the participant reports were analysed with regard to whether they
reflected pragmatic attributes of an interactive system, such as usefulness and
usability, and/or hedonic attributes, such as facilitating evocative or
stimulating experiences.
In the user reports, we found an appreciation for both pragmatic attributes
and hedonic attributes in participants' detailing of positive chatbot user
experiences; 42% of the participant reports reflected pragmatic chatbot
attributes and 36% highlighted hedonic attributes. In addition, 20% of the
reports reflected codes that are not directly related to pragmatic or hedonic
attributes. The most frequent of these additional codes concerned the social
aspects of an interaction (7%) and the chatbot's character as humanlike (4%).
Pragmatic attribute: help and assistance (34%)
The participant reports strongly reflected the importance of perceived
usefulness or practical value for positive user experience. When asked to
report a particularly good episode, participants often reported on getting
assistance or help from the chatbot. A number of the reported episodes
concerned customer service support and also instances of training or coaching
through the chatbot. Other episodes concerned personal assistance, such as
setting reminders for tasks or getting help with a specific task at hand, as in
this quote:

52
I asked what good places there was around me to eat and it brought up a list
and i chose from it. Now the place is one of my favorite places to eat at (P21)
The instrumental or pragmatic characteristics of chatbots were clearly
apparent in the reported episodes, where task achievement and efficiency in
particular were highlighted as important in numerous participant reports.
Participants reported receiving help in situations where they were pressed for
time because of an urgent problem or a circumstance where they needed
information quickly. Participants also made particular note that the assistance
was efficient and easily accessible, as in the following example:
The chatbot for customer support for my wireless carrier was a great help! I
didn't have to wait on a representative to become available, I was able to find
out what I needed to know about different plans and their pricing. So much
better than sitting on hold or waiting an hour for someone to message me back
(P23)
Pragmatic attribute: information and updates (8%)
While help and assistance for a particular task was by far the most frequent ly
reported category of positive user experience episodes, some participan ts
instead reported on the pragmatic benefit of chatbots for general informat io n
searches or more routine updates, such as news reports and weather forecasts.
These participants highlighted the chatbots' support for retrieving general
online information or daily updates, rather than getting help in a particular
situation. Participants reported gathering information through a chatbot that
they would otherwise be able to access through a search engine. Participants
also reported using chatbots for doing their everyday checks of informat io n
important to their daily routines, for example in the following quote:
I use google assistant to do simple tasks on my phone every morning when I
am waking up and need to know the time and weather […] (P59)

53
Hedonic attribute: entertainment (29%)
In their reports, participants indicated substantial appreciation for hedonic
chatbot attributes. When reporting on such non-pragmatic aspects of chatbot
user experience, participants typically highlighted the entertainment value of
chatbots. Entertaining chatbot episodes were presented in ways that indicated
they were seen as stimulating and contributed to the participant feeling happy
and engaged. Participants used words such as ‘fun’, ‘entertaining’ and ‘cool’,
as in the following example:
It was funny. I asked it if it liked me and it asked me if I like me (P207)
Participants who reported the entertainment value of chatbots typicall y
referred to situations where they engaged in small talk with a chatbot. That
is, they often did not have a particular task to be resolved but rather saw the
chatbot as a means of involving themselves in a pleasing activit y.
Specifically, they reported that the chatbots’ ability to be funny and witty was
a source of pleasure, or they reported that the chatbot was something they
could joke with or turn to when bored. An example of such use is reflected in
the following quote:
Chatbot and I just kept talking random things, that when looked at after made
some sense. it was fun (P128)
To our surprise, quite a few participants who reflected on hedonic aspects of
the user experience reported on the use of chatbots by children. Some of these
reports were from their own childhood; for example, participants reported
that conversations with chatbots as a child were a source of entertainment in
the company of their friends, or a source of relief as a teenager when they
were bored. Other participants described experiences as parents—observing
their own children engaging with chatbots, either on the initiative of the
parent or through the child's own initiative. The finding that chatbots serve
as a source of stimulation and engagement for children is interesting, as it

54
suggests the potential of chatbots to stimulate playful social interaction for
and possibly also among children.
My earliest memories of artificial intelligence are with an online chatbot
called SmarterChild. I remember it being pretty funny sometimes, witty and
intelligent, almost like it was a real person behind the character typing his
responses (P132)
Hedonic attribute: inspiration and novelty (8%)
Some of the participants who highlighted hedonic chatbot attributes in their
descriptions of good chatbot experiences reported an inspirational episode or
a general sense of novelty in chatbots.
Among the participants who reported on the chatbot as inspirational, some
described the episode as ‘eye-opening’ or described how they were able to
talk to the chatbot about a topic that engaged them, such as pets or food. Such
reports in part reflect the potentially evocative character of chatbots and in
part reflect the potential of chatbots to adapt to topics with which the user
identifies, as in the following participant quote:
I had a pleasant conversation about my life with a chatbot. I talked about my
family and my feelings (P182)
Some participants also reported being excited or engaged by their perception
of chatbots as a novel and fascinating way to interact with computers. In these
reports, some participants explained how they saw it as amazing to actually
have a conversation with a computer, and some also described how they had
tried to test the degree to which the chatbot is able to act like a human. The
following quote exemplifies participant reports belonging to this coding
category:

55
I had an interesting experience trying out artificial intelligence through small
talk with a chatbot. You could tell it wasn't human but it was interest ing
nonetheless (P83)
Other attributes: social and humanlike (11%)
While the coding categories for pragmatic and hedonic attributes are reflected
in most of the participants’ reports, some reports referred to attributes that
did not readily fit into the pragmatic- hedonic framework but were still
relevant to user experience. In particular, this was the case for participant
reports about the social value they received from using the chatbot (7%) as
well as the perceived experiential benefit of the chatbot being humanlike
(4%).
Social value typically involved enjoying a social situation with the chatbot.
In these reports, the participants described how they appreciated the social
interaction with the chatbot. That is, even though they were aware that the
chatbot is a machine, the social interaction was seen to hold value in itself,
as in the following example:
Chatbot helps me get my day moving when I don't talk to anyone (P141)
For some participants, the chatbot was used to support social interaction with
other (human) users, as for example in group chats. Here, the chatbot could
serve instrumental purposes in a social interaction, such as providing linked
content on topics of conversation or helping to get conversations or groups
started.
I used chatbots to send links to websites mentioned in Skype conversation. It
was very convenient way to make the conversation more efficient (P45)
The identified humanlike attribute was in many ways associated with the
social attribute. Here, participant reports explained how chatbot

56
characteristics that are almost human may contribute positively to user
experience. The humanlike character of chatbots was noted by some
participants, but fewer than might have been expected given that this is often
seen as a prominent chatbot attribute.
I use to ask them all kinds of questions till they had a whole conversat io n
with me told me where they were from an how they worked as a waitress at a
bar it was such a funny conversation i actually thought the chatbot was a real
person (P34)
In reports that emphasize the social or humanlike attributes of chatbots, these
attributes were often discussed together with hedonic or pragmatic attributes.
We nevertheless found it important to single out the social aspects of chatbots
as reflecting a distinct attribute outside the group of hedonic attributes
because the pragmatic-hedonic framework does not specifically address
social interactions with interactive systems. It should be kept in mind that our
coding scheme allowed any user report to have multiple codes associated with
it, so this represented no challenge in terms of coding.
Negative chatbot experiences
We asked the participants to report on poor or unpleasant chatbot user
experiences. In these reports, the most frequent characterizations of the
chatbot and the user experience involved pragmatic (23%) and hedonic (16%)
attributes, with pragmatic attributes being the most prevalent.
The distribution of codes for the participant reports on negative chatbot user
experiences is provided in Table 2. Details concerning each coding category
are provided following the table.

57
Limitations and future work

58
Limitations and future work
This study contributes insight into drivers of good and poor chatbot user
experience by gathering data from users of a highly diverse set of chatbots.
As such, the presented findings will be of interest for chatbot research and
development.
At the same time, the study has important limitations. First, whereas the study
allowed us to identify key aspects of chatbot user experience, the study
sample was not sufficient for a detailed breakdown of the relative importa nce
of these aspects for smaller user groups or for different chatbots or chatbot
types. This would make an interesting challenge for future survey studies with
larger samples. Specifically, we foresee more in-depth investigations into the
relative differences between task-oriented chatbots and chatbots oriented
towards social interaction with respect to how pragmatic and hedonic
attributes can be leveraged to improve user experience.
Second, whereas the study involved a broad range of users, they were all from
the same geographical region and language area. Replicating the study in
other regions may lead to different outcomes—among other things because
there is substantial variation in chatbot availability and support across
languages. It may also be argued that the participants in this study are
relatively more innovative and open than most other users with regard to their
technology use, given that chatbots are still an emerging technolo gy.
Specifically, what Rogers [40] refers to as early adopters may be more
prominent in this sample than they would be in a population where there is
widespread uptake of chatbots. As such, generalizing from the study find ings
to future populations of chatbot users requires consideration of their
similarities and differences with the population under study. We anticipa te
future research on chatbot user experience from the perspective of the
pragmatic-hedonic framework, which may shed light on how user experience
of chatbots evolve over time, regions and populations.

59
Third, whereas user experience is a subjective phenomenon, it also impacts
user behaviour. Hence, a comprehensive study of chatbot user experience
would benefit from also including chatbot user behaviour. We anticipa te
studies that combine large-scale questionnaire studies with data collection on
user behaviour, for example by using log data.

60
References
1. Bayan Abu Shawar and Eric Atwell, 2007 “Chatbots: Are they Really
Useful?”
2. LDV Forum - GLDV Journal for Computational Linguistics and
Language Technology.
3. http://www.ldv- forum.org/2007_Heft1/Bayan_Abu-
Shawar_and_Eric_Atwell.pdf
4. Bringing chatbots into education: Towards natural langua ge
negotiation of open learner models. Know.-Based Syst. 20, 2 (Mar.
2007), 177-185.
5. Intelligent Tutoring Systems: Prospects for Guided Practice and
Efficient Learning. Whitepaper for the Army's Science of Learning
Workshop, Hampton, VA. Aug 1-3, 2006.
6. http://en.wikipedia.org/wiki/Chatterbot
7. ALICE. 2002. A.L.I.C.E AI Foundation, http://www.alicebot.org/

61
Questionnaire
Name: _________________ Date: _________________
Address: _________________ Location:
____________________
Gender: _________________ Age: _____________
Q1) How is your experience with Chatbot?
A. Good
B. Very Good
C. Excellent
D. Bad
E. Very Bad
Q2) Would you recommend this company to others?
A. Yes
B. No
Q3) How is your organisation’s environment?
A. Employee’s Friendly
B. Stressful
C. Average

62
Q4) Chatbot is giving equal and fair treatment to all?
Strongly Agree
Strongly Disagree
Neither agree Nor Disagree

Saloni jaiswal mba-2nd semester_miniproject

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Saloni jaiswal mba-2nd semester_miniproject

Similar to Saloni jaiswal mba-2nd semester_miniproject (20)

More from RohanSilvenia

More from RohanSilvenia (7)

Recently uploaded

Recently uploaded (20)

Saloni jaiswal mba-2nd semester_miniproject