1.
Chatbots in HR
Improving the
Employee Experience
Kong Yean Hwei, Amy | Advanced Work Based Project Report
Crew 4 | Hyper Island Singapore | Jan 2017

2.
2
Table of Contents
EXECUTIVE SUMMARY ……………………………………………………………. 2
WHAT ARE WE EXPLORING? …………………………………………………….
Changing Landscape in Advertising ……………….…………….……………
Digital is Part of the Process ……………………………………….…………….
Why Chatbots, and Why HR? ……………………………………………………
4
4
4
5
EXPLORING THE LANDSCAPE – A LITERATURE REVIEW ………………
Origin and Development of Chatbots …………………………………………
Why are Chatbots of Such Interest Now? …………………………………….
Applications andBenefits ofHRChatbots ……………………………………
Chatbots in PubComms SG for HR Information Services ……………….
Designing and Building Chatbots ………………………………………………
Some Considerations When Implementing Chatbots ……………………
Learnings and Conclusions for an Experiment ……………………………..
6
6
7
9
11
11
14
15
THE HR CHATBOT EXPERIMENT ……………………………………………….
Experiment Considerations ………………………………………………………
Experiment Design ……………………………………………………………….
Experiment Round 1 with Twine – New Insights …………………………...
Experiment Round 2 with Chatfuel – Validation and More …………….
Overall Findings and Conclusion ………………………………………………..
Limitations of the Experiment …………………………………………………..
16
16
16
18
24
28
30
REVIEW AND RECOMMENDATIONS ……………………………………………
Yay or Nay? ……………………………………………………………………………
How Much Does It Cost? ………………………………………………………….
It Could Save SGD 125,000 Annually …………………………………………..
30
30
30
30
Suggested Timeline and Next Steps …………………………………………..
Other Recommendations and Conclusions ………………………………….
The Time is Now …………………………………………………………………….
30
32
32
REFLECTIONS ………………………………………………………………………….. 32
REFERENCES ..……………………………………………………………………..….. 35
APPENDICES ..………………………………………………………………..………..
Appendix 1: Key Questions to Ask Before Investing in a Chatbot ……
Appendix 2: Proposal to PubComms SG …….…………..……………….…
Appendix 3: Qualitative Interviewing Framework …..…..…………....…
Appendix 4: Twine Documentation on ERIS Prototype 1 .......………..
Appendix 5: Experiment Data and Analysis ………………………….……..
Appendix 6: Chatfuel Documentation on ERIS chatbot ………….……..
41
42
ABOUT THE AUTHOR
The author was most recently the HR Director in Publicis Communications
Singapore. She held dual roles, leading HR operational responsibilities for
Publicis Worldwide, MSLGROUP and Nurun, and the Learning & Development
function for all brands within Publicis Communications Singapore, including the
brands mentioned above plus Saatchi & Saatchi, Leo Burnett and Prodigious.
This Advanced Work Based Project is based on Publicis Communications
Singapore.
REDACTED
REDACTED

3.
3
EXECUTIVE SUMMARY
Technology has disrupted the advertising and communications industry.
Agencies are seeing a chronic decline in industry prices of 4.5% to 5% per
year, compounded. Companies are trying to capitalize on this digital wave,
including Publicis Groupe, the holding company of Publicis
Communications,which has set this as a strategicpriority for the company.
It is logical to assume that internal company processes should also reflect
these digital priorities, and this Advanced Work Based Project aims to be
part of the solution by exploring how technology applied to processes
within Publicis Communications Singapore can provide value.
Specifically, this report sets out to explore the following problem
statement: “How might we explore the potential and applications of
chatbots in HR, specifically in Publicis Communications Singapore, to
improve the employee experience?”
This report reviews the origin and development of chatbots, and the
confluence of recent developments that have triggered the surge of
interest in chatbots. It also reviews the potential applications of chatbots
in HR and its benefits, and considerations needed for the design,
development, and implementation of chatbots.
An experiment using the Build-Measure-Learn feedback loop of the Lean
Startup methodology was conducted in Publicis Communications
Singapore to gain employee feedback on the problem statement. Two
prototypes were tested with a small group of employees; one was a low-
fidelity prototype using the Twine program, and the other a chatbot
Minimum Viable Product (MVP) written using Chatfuel, a Do-It-Yourself
chatbot builder.
To access the ERIS MVP on Chatfuel, please email the author at
amy.kong@hyperisland.sg for an access link.
Based on the learnings of the literature review, the chatbot was developed
as a text-based, rules- and retrieval-based, closed domain chatbot with
some level of artificial intelligence. The topic of the chatbot was focused on
providing information on a specific HR policy.
Interviewees were asked to interact with the prototype, then semi-
structured interviews were conducted to get their feedback on the
prototype and their views on the usefulness and potential applications of
chatbots in the HR function in PubComms SG.
Overall results show that employees like chatbots’ ability to provide instant
access to information, the ability for them to gain answers independently,
and friendly tone. Each mentioned at least two other areas of applications
of chatbots in HR, including providing information on company policies,
processes, onboarding, and training.
Additional insights that emerged included that interviewees felt chatbots
would help them to avoid embarrassment when they needed to ask certain
types of questions, and that chatbots could provide anonymity when they
need to read up on sensitive company policies.
Interviewees were clear that they would go to a chatbot for functional
queries, while they would go to a HR team member for counsel on non-
routine requests or emotional issues. This opens the possibility that
chatbots could help to transition HR away from a transactional role towards
an expertise role in the company.
Based on the results of the experiment, there is significant potential and
value in investing in HR chatbots to improve the employee experience and
employee productivity. Calculations show that Publicis Communications
Singapore could save about SGD 125,000 annually if just 5 minutes were
shaved off each HR request from employees.

4.
4
WHAT ARE WE EXPLORING?
Changing Landscape in Advertising
Technology has disrupted most industries, and it is no
different in advertising. Where advertisements
previously lay predominantly in the domain and
control of advertising agencies, the advent of
technology has changed that.
The industry has shifted dramatically and advertising
agencies now face challenges on multiple fronts:
• Emergence of new forms of advertising such as
online, mobile and search engine marketing (SEM)
that has seen a redistribution of budgets from
traditional forms of advertising (Pandey, 2016)
• Competition from companies that were never
considered competitors before, such as
o Management consultancies (Dan, 2016;
Roxburgh, 2016; Vranica, 2016)
o Publishers (Marshall and Alpert, 2016)
o Digital tech companies (Johnson, 2015)
o In-house agencies (Schaefer, 2015; Stiglin,
2015; Morrison, 2016)
• Commoditization of the industry as technology
enables anyone to become creative content
creators (Fromowitz, 2016; Oetting, 2015)
As a result, the industry is seeing huge pressure on
prices and margins. According to Farmer (2015),
declining fees (at 2-3% annually) and growing
workloads (2-3%annually)overthepast 20years have
resulted in a chronic decline in industry prices of 4.5%
to 5% per year, compounded.
Digital is Part of the Process
Many companies, and industries, are trying to
capitalize on this digital wave, including Publicis
Groupe. Publicis Groupe is the holding company that
owns Publicis Communications (PubComms), the
Client brand that this Advanced Work Based Project
(AWBP) is based on.
In fact, it is of such a strategic priority to Publicis
Groupe that it separately reports its digital revenues
in its quarterly earnings reports, and current digital
revenues stand at 54% of total Groupe revenue in the
third quarter of 2016 (Publicis Groupe, 2016b).
It is also investing in training and development efforts
for its employees. In its 2015 Annual Report
Registration Document (Publicis Groupe, 2016c), it
states, “…the goal is to ensure that each employee is
able to acquire basic know-how, whether in
rudimentary coding or better understanding the
latest generation applications. The boom in mobile
device usage and the new challenges of
interconnectivity (connected objects) have led
agencies to hire and train talented individuals with
multiple skills, given the speed of industry change… It
is essential to support our future managers.”
If employees are expected to be digitally-savvy and
able to recommend leading edge digital
communications solutions for clients, then it is logical
to assume that internal company processes should
reflect these digital priorities as part of the digital
immersion for company employees.
According to Edgar Schein (cited in Christensen et al,
2016),processes are a criticalpart ofan organization’s
unspoken culture. They tell people inside the
company, “This is what matters most to us.”
However, there are some that say that despite all the
rhetoric, things are not changing:
“What’s broken is not our people. It’s our process. It’s a process
that has been endlessly debated but not reinvented,
and it has not adapted to the changing world around us.”
Grayson and North, 2016

5.
5
“What’s broken is not our people. It’s our
process. It’s a process that has been endlessly
debated but not reinvented, and it has not
adapted to the changing world around us... We
traffic in risky ideas. But we don’t make any
bold moves to change how we operate
internally.” (Grayson and North, 2016)
This Advanced Work Based Project (AWBP) aims to be
part of the solution, by exploring how technology
applied to processes in a company – Publicis
Communications Singapore (PubComms SG) - can
provide value. Specifically, the use of chatbots in HR.
Why Chatbots, and Why HR?
Chatbots are a hot topic now, but most of the current
discussion seems to focus on the use of chatbots to
engage with customers. One of the functions of
Human Resources (HR), as a parallel, is like the
employee customer service function of the company.
So, would it be a far stretch to say that chatbots could
also have significant potential in HR?
Currently, most of the software or technology used in
the PubComms HR function are used for non-
employee facing activities (backend functions) to
ensure operational continuity or as a management
reporting tool, e.g. payroll systems, PTalent (Publicis
Groupe’s HR Information System). There are
currently few technologies used that facilitate or
assist the interaction between the HR function and
employees, and all interactions are manually
managed, e.g. HR queries, notifications to employees
on HR updates etc.
Improving the employee experience cannot be
underestimated. Advertising agencies are expertise-
led companies, i.e. talent is a strategic differentiator,
and so talent attraction and retention should be key
activities for the company. This is even more
important now, as new competitors like tech firms
and startups are aggressively wooing talent away
(Tadena, 2015; Ember, 2016).
This represents an area of opportunity for exploration,
to use technology to improve processes and help
create a company culture that is aligned with its
strategic priorities, that in turn improves the overall
employee experience.
As such, the defined problem statement that we will
use in this paper is:
“How might we explore the potential and
applications of chatbots in HR, specifically in
PubComms SG, to improve the employee
experience?”
ABOUT PUBLICIS COMMUNICATIONS
BACKGROUND
PubComms is one of 4 Solutions hubs within Publicis
Groupe and comprises all of Publicis Groupe’s creative
brand networks, including Publicis Worldwide,
MSLGROUP, Nurun, Saatchi & Saatchi, Leo Burnett,
BBH, Marcel and Prodigious (Publicis Groupe, 2016a).
Publicis Groupe is one of the largest marketing and
communications company in the world, alongside
WPP, Interpublic, and Omnicom (Elliott, 2002).
In early 2016, a Chief Executive Officer was appointed
to lead Publicis Communications in Singapore
(Mumbrella Asia, 2016), and subsequently a Chief
Talent Officer and a Chief Finance Officer. The finance
and HR functions across the various brands were
consolidated to align with the new management
structure, overseeing a headcount of about 250
people.
As each creative brand previously had its own HR
policies, the HR team (author included), was tasked to
align and improve employee and HR operations and
policies across the various brands under PubComms
SG.
CLIENT STAKEHOLDERS
The key client contact for this project was Dan
Spencer, Chief Talent Officer, Singapore and
Australia/New Zealand, Publicis Communications.
Other key stakeholders included the HR leaders of
PubComms SG, Sharon Ooi, HR Director, and Adele
Sam, Talent Manager, who provided the information
and liaison support in the execution of the
experiment.
REDACTED
REDACTED
REDACTEDREDACTEDREDACTED

6.
6
EXPLORING THE LANDSCAPE – A LITERATURE REVIEW
Origin and Development of Chatbots
What is a Chatbot?
Different terms have been used for chatbots (or
chatterbots), including conversational agents,
dialog systems and virtual assistants. This is due to
the proliferation of a variety of similar systems built
with different technical architecture (Perez-Marin
and Pascual-Nieto, 2011), for different purposes.
For this report, we define a chatbot as an automated
online software program that tries to mimic a
human conversation in its interaction with a human
user to answer questions or perform tasks (Shawar
and Atwell, 2007; Knowledge@Wharton, 2016).
Specifically, we will focus this report on text-based
chatbots.
Chatbots can be built for websites or apps and
accessed through smartphones, tablets, desktops
and tablets. In recent years, most of the major
messaging platforms have launched tools to help
developers integrate or deploy chatbots on their
platforms.
These include Facebook Messenger, Slack,
Telegram, Google and Microsoft, Kik, among others
(Richman, 2016).
What makes a Chatbot?
For a chatbot to conduct human-like conversation,
ArtificialIntelligence (AI) is a criticalcomponent.Nils
J. Nilsson, as cited by Stone et al. (2016), defines AI
as the “activity devoted to making machines
intelligent, and intelligence is that quality that
enables an entity to function appropriately and with
foresight in its environment”.
AI has many applications (Stone et al., 2016), but in
this report, we will narrow the topics to those
relevant to text-based chatbots.
Chatbots functions in two ways (Schlicht, 2016):
1. Rule-based: These types of chatbots respond to
very specific commands (rules), and are unable to
process if information is not input in the right way.
It is only as smart as you program it to be
2. Machine Learning: Machine Learning is a form of
AI, and machine-learning chatbots continuously
get smarter as they learn, via Natural Language
Processing (NLP), from the conversations it has
with people
The recent boom in chatbots has been fuelled by
advancements in AI research, combined with
technological improvements such as the availability
of cloud computing resources and wide-spread,
web-based data gathering (Stone et al., 2016).
These advancements in AI include the areas of:
• Natural language Processing (NLP) and Natural
Language Understanding (NLU)
NLP explores how computers can understand
and manipulate natural language text or speech
to do useful things (Chowdhury, 2005). NLU is a
subset of NLP that deals with machine reading
comprehension (Ovchinnikova, 2012), i.e. the
ability of the computer to understand human
language. In short, NLP applications try to
understand natural human communication and
communicate in return (Marr, 2016).
• Machine Learning
With machine learning,computers areenabledto
continuously learn from data on its own, using
algorithms (set of rules or steps), to find insights
without being explicitly programmed where to
look (Sas.com, n.d.). In chatbots, machine
learning is used to help machines learn and
understand the vast nuances in human language,
and learn to respond in a manner that the
audience is likely to comprehend (Marr, 2016).
While this report will not go into the details of these
technologies, these are key terms to understand as
they will be mentioned in any discussion around
chatbots.

7.
7
Key Developments in AI and Chatbots
Turing Test and Loebner Prize Competition
One of the most significant milestones was created
when Alan Turing created “The Imitation Game”,
now known as the Turing Test, in 1950.
Starting with the question “Can machines think?”
(Turing, 1950), Turing created a test that would
determine if a computer is capable of thinking like a
human. The Turing Test has since been used as a
benchmark that any AI must pass en route to true
intelligence (Ball, 2015).
However, it was not until 1991 that there was an
implementation of the Turing Test (Loebner.net,
n.d.b) – the annual Loebner Prize Competition –
created by Dr. Hugh Loebner to advance the field of
AI (Loebner, n.d.a). As long as there is an entry for
that year, the prize will be awarded, and so while
there have been multiple Loebner Prize winners, no
program passed the Turing Test until 2014 (Griffin,
2014).
The First Chatbot, ELIZA
Then in 1966, the first chatbot ever coded, ELIZA,
was documented by Joseph Weizenbaum. ELIZA
was written to simulate a psychotherapist
conducting a psychiatric interview (Weizenbaum,
1966). ELIZA worked by analysing the words users
inputandmatching themto a list ofpossible scripted
responses (Newman, 2016).
More critically, ELIZA seemed so believable to users
that Weizenbaum perceived his program as a threat,
and in his book Computer Power and Human Reason,
he attacked AI (specifically ELIZA) and computer
science research, thereby slowing the pace of
research into AI (Epstein, Roberts and Beber, 2008).
A.L.I.C.E
In 1995, A.L.I.C.E. (Artificial Linguistic Internet
Computer Entity), the first AIML-based (Artificial
Intelligence Markup Language) personality program,
was created.
AIML is a computer language designed for creating
natural language-based chatbots that is still used
today.
A.L.I.C.E.won theLoebnerPrize three times,in 2000,
2001 and 2004 (Epstein, Roberts and Beber, 2008),
and is one of the strongest AI programs of its time.
It was programmed to give the illusion that it was
intelligent and self-aware, but was driven by
“supervised learning”, where a botmaster monitors
the conversations and creates content to make
responses more appropriate, accurate, “human”, etc
(Epstein, Roberts and Beber, 2008).
Bots for Messenger
In April 2016, Facebook launched the bots for
Messenger Platform (Marcus, 2016). While
Facebook is not the first company to launch a bot
platform, it is significant because it now provided, at
scale, the opportunity for businesses to automate
their one-to-one engagement with their customers
to drive business and e-commerce (Rosenberg,
2016), thus driving the current interest and boom in
chatbots.
Figure 1 shows a more detailed timeline on the
history of chatbots.
Why Are Chatbots of Such Interest Now?
A confluence of several developments has triggered
the surge of interest in chatbots:
Advancements in AI Research and Computing
As mentioned, advancements in AI research,
particularly in machine learning and NLP, are
enabling chatbots to provide more accurate and
intelligent responses.
Improvements in computing and hardware
technologies have also enabled integration of
various apps into a chatbot that allow chatbots to
complete tasks with little or no operator
intervention.
In fact, research firm Gartner predicts that in 2017,
only 33% of all customer service interactions will
need a human intermediary, as compared to 60% in
2014. (Gartner.com, 2015)

8.
8
Figure 1.: The History of Chatbots (Futurism, n.d.)

9.
9
Rise of Mobile Messaging and Fall of Apps
According to Business Insider (2016), monthly active
users of messaging apps (where chatbots can reside)
surpassed social networks in 2015 in terms of
monthly active users (see Figure 2).
Companies also face increasing friction in getting
consumers to download and use a mobile
application. The average mobile user globally has 33
apps installed on his or her device, of which 12 apps
are used daily. Of that, 80% of the average global
mobile user’s time is spent only on 3 apps (Meeker,
2016).
This means that companies and brands are finding it
increasingly difficult to engage one-on-one with
customers.
App Integrations Attract Users to Stay Within
Platform Eco-system
The ability to integrate various apps into a chatbot
also means that users are attracted to stay within
the messaging app (Bayerque, 2016).
In the context ofbrands,it means that customers are
likely to stay within the brand eco-system, thereby
increasing duration of interaction with the brand.
Bots Can “PiggyBack” Existing Massive Platforms
Bots are the beginning of micro apps on the backs of
massive platforms which will lead to more focus and
reach for start-ups and more delighted users
(Batalion, 2016) because brands can:
1. Spend less time in development, e.g. building a
chatbot on Facebook Messenger allows you to
build one experience (without needing to write
for different mobile phone sizes, or operating
systems) and access Facebook user rich profiles
instantly
2. Focus more time on creating positive user
experiences because they can leverage richer
user interfaces and software plug-ins (that
platforms develop or offer) to process input
What all this means is that companies and
organisations can now increase quality one-to-one,
personalised engagements with customers at scale,
at potentially reduced costs and time.
Applications and Benefits of HR chatbots
Current Applications of Chatbots in HR
General HR Support
Information Services (Search and Retrieval)
A typical challenge that employees face is knowing
where and how to find information on HR policies
and processes, or even whom to approach within the
HR department for specific queries.
Benefits of using chatbots in information services
include (McNeale and Newyear, 2013):
• Tackling the problem of users not familiar with
specific terminology and jargon
• Unlike humans, chatbots are unruffled by rude
customers, high traffic, or repeated requests for
the same information, and remain consistently
patient and polite
This also shifts the responsibility of locating the
needed information from the user to the designer of
the chatbot where the designer leads the user
through a question and answer dialogue to discover
the information needed and provide it (McNeale and
Newyear, 2013).
An example of such a chatbot is “Ask Ivy”, a “virtual
HR agent” launched by Intel in 2013 that uses a
combinationofNLP,AIandoptimizedsearchhosted
on their Intranet to help answer employees'
questions about their pay, stock, benefits, or other
HR programs (Pearce, 2013).
Figure 2.: Messaging Apps Surpass Social
Networks (Business Insider, 2016)

10.
10
HR Task Automation or Assistance
Virtual personal assistants are now a big focus due
to the ability to now integrate several apps (and
actions) into chatbots.
The most current and well known examples are
iPhone’s Siri, Amazon’s Alexa, Microsoft’s Cortana
and the Google Assistant (Dunn, 2016).
In the domain of virtual chatbot assistants in HR,
there are some products now in development in the
market, including:
• HR task automation chatbot from ADP
Innovation labs that is still in development
(Boulton, 2016)
• Mila, overstock.com’s chatbot created to assist
with employee sick leave reporting and team
rescheduling at their call center (Greenfield, 2016;
Aspect.com, n.d.)
Recruitment
Streamlining Job Applications
Chatbots can be used to vet potential candidates
(Bhaduri, 2016). One recent example is Mya
(http://trymya.io/), a chatbot that vets and
interviews job candidates, using textual analytics to
filter irrelevant resumes and interviews selected
candidates on their work experience and
qualifications. Mya also administers tests, provides
application status updates and tips to candidates
(Dishman, 2016; Deutscher, 2016; Trymya.io, n.d.).
Q&A Tool for Job Applicants
Chatbots can also serve as a Q&A resource for
potential candidates on the position they are hiring
for, or the company’s hiring policies (Sullivan, 2016).
Benefits of using chatbots this way include: Higher
quality of applicants (through candidates’ self-
selection), improved candidate experience, reduce
time spent by recruiters (and costs) in answering
repeated FAQs from candidates.
One of the most established examples is Sgt. Star,
launched in 2007 by the United States Army to help
potential recruits to learn about a career in the Army.
First residing on the Army recruitment webpage,Sgt.
Star then made its debut as a mobile app in 2014.
(Maass, 2014; Nextit.com, n.d.).
Onboarding
Chatbots can also beusedto contactnew employees
(or allow them to self-serve) with onboarding
information, such as organisational structure and
history, how to get enrolled in programs or benefits,
suggestions on colleagues to meet, and links to the
required HR forms, in place of a HR generalist
(Boulton, 2016; Zwier, 2016).
Training (Bhaduri, 2016)
There is already ongoing discussion in academic and
education sectors on the value of chatbots in
training and learning that can be applied to the
corporate sector (Perez-Marin and Pascual-Nieto,
2011; Olson, 2016). Chatbots can be used to deliver
interactive and personalised training, monitoring
and nudging, and instant help (Newton, 2016; Riel,
2016).
Benefits of Using Chatbots in HR
Based on the above, the potential benefits of using
chatbots in HR include improving:
• Employee productivity and employee
experience by:
o Providing on-demand employee
engagement with 24/7, instant responses
o Simplifying information access
o Supplying just-in-time and fast access to
information
o Personalisation of the employee interaction
• HR team’s performance (value-add, productivity
and motivation) by
o Handling repetitive tasks (e.g. answering
FAQs) thereby
- Freeing up HR resources to focus on
more value-added work or reduce costs
- Reduce frustration and/or monotony of
the HR team
o Automating standardised tasks
o Identifying potential issues using data
analytics and addressing them before they
escalate

11.
11
Chatbots in PubComms SG for HR
Information Services
The experiment conducted to explore the problem
statement of this report focused on the use of
chatbots to provide HR information services in
PubComms SG as it strikes the best balance
between feasibility and desirability.
Why Information Services?
Firstly, the HR function in PubComms SG already
has ready content that can be re-purposed to
populate any chatbot prototype.
Secondly, as a first investment in chatbots, the
easiest chatbots to build are rules-based chatbots
that rely on specific topics (closed domain) with
predefined responses (retrieval-based) (Figure 3.).
Having machine learning capabilities is a bonus and
can be incorporated into the chatbot with
subsequent iterations (Varagnat, 2016).
Jumping into a chatbot that requires extensive
machine learning capabilities from the start will
create barriers of entry as it can be time-consuming
(due to lack of expertise) and potentially costly.
What About the Other Applications of
Chatbots?
While recruitment is a constant needfor PubComms
SG, it is not feasible because:
• In the communications agency environment,
the company information that candidates
would be interested in (e.g. client roster) is
considered confidential. It is not available on
public forums and usually only communicated
face-to-face to the candidate verbally during
the interview. The highly customised nature of
remuneration in the advertising industry also
does not suit the use of chatbots for recruitment
Q&A at this point in time
• Using chatbots to vet potential employees
require sophisticated filtering technology as the
interviewing process also involves interviewing
for soft skills. The technology is still very new,
and to demonstrate the potential of chatbots in
PubComms SG, it is better to choose an area
that is more established. Additionally, the
various hiring managers within PubComms SG
have different interviewing and selection
criteria, and it would be difficult to standardise
this quickly to build the prototype. Finally, most
off-the-shelf products in the market are US-
based and there may be cultural differences in
the way resumes are written, or how candidates
respond in an interview.
In the area of HR task automation, companies
openly admit that it is very early days and all these
assistants are far from polished (Dunn, 2016).
As for onboarding and training, there are currently
no consolidated company information materials
(due to the recent re-organisation) that can be used
to populate the chatbot. Use cases of the value of
chatbots in corporate training in the industry are
also still few.
Designing and Building Chatbots
This section is split into the following subsections,
to explore best approaches for designing and
building chatbots:
1. What Consumers Want
2. Three Approaches to Creating Chatbots
3. User Experience Design Consideration
REDACTED

12.
12
Appendix 1 also provides a list of key questions to
consider prior to investing resources in a chatbot.
What Consumers Want
Prior to designing a chatbot,it is worth exploring the
functions and experiences consumers want. As this
is a relatively new area of interest commercially,
there are no any large scale, benchmark global
consumer surveys that have been conducted, but
some surveys have been conducted in specific
markets. These provide starting insights that could
be validated through Proofof Concept experiments.
Based on results of three recent consumer surveys,
one in the US and two in the UK with sample sizes of
1,000 respondents each (Aspect, 2016; Mindshare,
Goldsmiths University of London, 2016; myclever
Agency, 2016), these are some of the findings
grouped around certain central themes:
“Humanness” of a Chatbot
Consumers prioritize efficiency in understanding
their queries over the personality or friendliness of
the chatbot. This was raised in the two UK surveys.
However, the US survey indicated that consumers
expect chatbots to perform better in “friendliness”
and “ease of use” as compared to “accuracy” and
“interaction success”. These could be inter-related,
i.e. consumers don’t have a good experience with
the efficiency of chatbots, therefore prioritising
them higher than the tone of interaction of chatbots.
Expected Benefits
• Getting 24-hour service
• Quick answers to simple to moderate questions
• Getting an instant response
• Being able to self-serve without having to talk to
a customer service agent
Predicted / Expected Functions of Chatbots
• Quick emergency answers
• Forwarding to appropriate human when
requested by user
• Buy basic items (clothes, food)
• Retain information on context and history of all
previous interactions for a more personalised
experience
• Get basic information about a company, product,
or service
• Information on product availability
Three Approaches to Creating Chatbots
There are three ways to create chatbots (Vorobiov
and Kotsurenko, n.d.):
1. Buy a ready solution: there are vendors who
offer off-the-shelf solutions. Examples of HR-
specific chatbot solutions include Mya,
mentioned earlier, Talla for onboarding
employees within Slack (https://talla.com/). This
allows businesses to speed up time-to-market,
but the downside is cost and not all ready-made
solutions are customisable.
2. Use DIY chatbot builders: there are a variety of
Do-It-Yourself (DIY) bot builders, some requiring
technical knowledge, and others not requiring
any. A higher degree of customisation is possible,
and it is relatively cheap and quick. Functions
may be limited but new features are constantly
being added.However,it can be hardto compare
functionalities and prices across chatbot builders
as business models vary widely.
3. Build a chatbot from scratch: While technically
still the most challenging, there are many
developer tools available in the market that
make chatbot building much easier now.
Naturally, this option requires technical
expertise and more cost, but results in a solution
that is tailor-made for, and proprietary to, the
company.
Figure 4 shows an infographic ofthe current chatbot
landscape. This infographic does not include
chatbot development vendors as they are industry-
specific, but it gives a good overview of the market.
As the objective of this report is to explore the
potential and applications of chatbots in HR to
improve the employee experience, we will focus on
using DIY chatbot builders (referred to as chatbot
builders from here) as it is the cheapest and fastest
way to build a prototype, and evaluating technical
considerations is not an area of discussion here.

13.
13
User Experience Design Considerations
Given that part of this report’s problem statement is
concerned with improving the PubComms SG
employee experience, we need to examine how to
create good User Experience (UX) in a chatbot.
Thisis especially whenchatbotsnowenableone-on-
one interactions between the user and the company,
the quality of that interaction becomes more
important.
Many articles use the terms UX and User
Interface (UI) interchangeably. For clarity, we
attempt to make a distinction between the two. The
industry perspective of UX focuses on the
“optimisation of a product for effective and
enjoyable use”, including ensuring the product
logically flows from step to step (Lo, 2014;
Lamprecht, 2016).
UI, on the other hand, is concerned with the “look
and feel, the presentation and interactivity of the
product”that guides theuser intoa goodinteraction
with the product.
To simplify it even further, the comparison between
UX and UI:
“Something that looks great but is difficult to
use is exemplary of great UI and poor UX.
While something very usable that looks
terrible is exemplary of great UX and poor UI.”
Helga Moreno, as cited by Lamprecht (2016)
In this context, a large part of creating a good UI is
already handled by chatbot builders as their
products come with pre-set user interfaces. Some
industry best practices to designing a good UX in
chatbots (Ady, 2016; Howard, 2016; Jain, 2016; Yao,
2016):
Smallest Number of Steps
Plan the conversation flowto be efficient,so that the
user reaches his goal in a few steps as possible.
Always Provide a Path to the End Goal
Give users cues to reach next steps that lead to the
end point, so that they are not left hanging mid-way
in a conversation without knowing what to do.
Figure 4.: Chatbots Landscape (Medium, 2016)

14.
14
Decide Parameters of Information to be Stored
If one of the aims of the chatbot is to provide a
personalised experience, then information about
the user needs to be retained / stored. Establishing
parameters (or categories) of information to be
stored helps to provide targeted or relevant
information for users. For example, if there are
different sets of employee benefits associated with
different job titles in the company, establishing job
titles as a parameter allows the chatbot to retrieve
the different sets of employee benefits by the user’s
job title.
Set the Right Expectations
• Explain the purpose of the chatbot
• Explain what the chatbot can do
• Explain how to interact with the chatbot
• Set a first action for the user to take
• Provide hints or prompters on the possible paths
of exploration at the end of the onboarding
Allow Flexibility of Exploration
Allow users the flexibility to skip around options.
Use Buttons to Supplement Free Text Input
Particularly for closed domain chatbots, buttons
provide more accuracy in interpreting users’
requests since the options are pre-defined. Clicking
a button is also faster than typing text.
Layer AI to Interpret User’s Requests
Use AI to learn the various ways users phrase
requests differently so that the chatbot can trigger
the right response or action, e.g. “Hi”, “Hello”,
“Wassup”, “Yo” should trigger a return greeting or
opening message from the chatbot.
Use Images or Graphics to Supplement Text
Images or graphics are easier to understand than
text explanations.
Provide a Null Response
If the chatbot is unable to understand what the user
wants,a message shouldbe triggeredto indicate so.
Non-responses from the chatbot where the user is
left not knowing what to do next should be avoided.
Tone Should Reflect the Brand Personality
For example, the chatbot may want to use more
casual, trendy words if it is targeted at millennials.
Also consider the use of emoji or gifs or graphics
appropriate to the chosen tone of voice.
Help Users Get Help
Provide options for users to get help outside of the
chatbot as needed. This can include directing the
user to documentation, or a human point of contact
for further inquiries.
Integrate with Existing Business Systems
Enable chatbots to access CRM databases to allow a
richer experience with users and avoid having the
user repeat their history or user details every time
they interact with the chatbot.
Some Considerations When
Implementing Chatbots
Hype
There is a lot hype currently around chatbots, such
that the general public's expectations of what
chatbots can do will exceed the reality of what they
can actually do (Hobson, 2016).
This means that in launching any chatbot,
companies should take care to communicate clearly
to users what they should expect from any
interactions with that specific chatbot.
The reality also is that not every bot needs to be
sophisticated, and it will depend on the objective or
outcome that the chatbot is built for.
Ifit is meant as an informationalservice,there is very
little point in building a bot to conduct a
conversation with a user, when the bot is meant to
be transaction-based (Newman, 2016).
Companies that are considering implementing
chatbots need to be clear about the objective or goal
of the chatbot and evaluate the technologies or
functions needed to achieve that, rather than
include unnecessary user features that result in
additional cost for the company.

15.
15
Benchmark for a Minimum Viable Product (MVP)
Software companies are used to developing a MVP,
or a minimum standard version of the product that
consumers would be willing to buy, to test in the
market.
However, for a good user experience, the product
will likely need to have more accurate NLP and
information before a MVP can be developed, which
may mean that chatbots could require more capital
than a traditional web or mobile app, where good
frameworks are more commonly available (May,
2016).
As above, clarity in the goal of the chatbot is
essential,to be able to strip down to a MVPfor users.
Alternatively, companies can first pilot a chatbot in a
very confined and specific area to minimise capital
outlay, while not compromising on the UX of the
chatbot. This is the approach that has been adopted
in developing the chatbot prototype for PubComms
SG, as seen in the next sections of this paper.
Learnings and Conclusions for an
Experiment
Chatbots are not new, but advancements in AI and
computing technology have enabled the
development of chatbots that can understand users
better, have more meaningful conversations and
perform tasks more effectively and accurately. This
has opened chatbots for commercial possibilities at
scale,particularly chatbots on messaging platforms.
There are multiple possibilities of applying chatbots
in HR; specific to PubComms SG, it is recommended
that the focus of any chatbot pilot be focused on
providing information services on specific topics
that allow a MVP to be developed quickly and cost-
effectively for piloting with employees.
Also, the best approach in creating one seems to be
the use of chatbot builders to minimize cost,
increase the speed of prototype development and
remove the constraints of needing technical
expertise. It also allows more effort to be focused on
providing a quality user experience to help
employees better visualise the value and
possibilities of using chatbots in the HR function of
PubComms SG.

16.
16
THE HR CHATBOT EXPERIMENT
Experiment Considerations
Demonstrate Proof of Concept (POC)
PubComms SG has never implemented similar
technology in its support functions, so it was
important to gain enough feedback to demonstrate
POC to the PubComms SG HR team for them to
evaluate if resources should be invested in such a
tool, and provide justification to higher
management on its value.
Lean Resources to Build the Experiment
As there was no budget, the experiment would have
to be built on free platforms.
Platforms That Do Not Require Much (or Any)
Coding Knowledge
As the author has no background in coding, the
chatbot would have to be built on tools that require
little to no coding knowledge. As the objective of
this experiment was to gain employees’ feedback,
and not evaluate technical capabilities of chatbots,
this would not pose a significant problem in the
experiment.
Topic where a Chatbot would Provide Noticeable
Value
To demonstrate the value of a chatbot to users, the
topic selected would need to be one where it is not
common knowledge, or of a complexity level that
would require employees to seek information or
help.
Ease of Testing and Confidentiality
As PubComms SG currently has no intranet, the
experiment would have to be designed and hosted
on external (and possibly public) platforms. As such,
the topic selected for the experiment should have
no confidentiality impact on PubComms SG.
Speed
The chatbot had to be created quickly as the
experiment neededtobe builtandconductedwithin
one month.
Experiment Design
Parts of the Lean Startup methodology (Ries, 2011)
was used to design the experiment. Advantages of
this methodology vs traditional research or product
development methods include developing products
that customers (employees) want, more quickly and
cheaply, and making it less risky (Blank, 2013).
If we were to view the proposed HR chatbot as a
startup, there are three distinct stages (Figure 5.)
and two periods of focus (Maurya, 2012):
Stage 1: Problem/Solution Fit – Is there a problem
worth solving?
Stage2:Product/Market Fit – HaveI built something
people want?
Stage 3: Scale – How do I accelerate growth?
The experiment was designed to explore and
answer Stage 1 and 2; i.e. show validated learning
that will help with presenting recommendations to
the client.
PROBLEM / SOLUTION
FIT
•
Stage 1
PRODUCT/MARKET
FIT
••
Stage 2
SCALE
•••
Stage 3
Focus: Validated Learning
Experiments: Pivots
Focus: Growth
Experiments: Optimizations
Figure 5.: Three Stages of a Startup (Maurya, 2012)

17.
17
Build-Measure-Learn Feedback Loop
A key component of Lean Startup is the Build-
Measure-Learn Feedback Loop (see Figure 6.) The
aim of this is to go through the Feedback Loop in as
little time and as few resources as possible to
maximize learning through incremental iterative
engineering and gain insights (Blank, 2015).
It would allow the requiredminimum feedback to be
gathered to demonstrate POC, enable the
experiment to be conducted quickly, enable pivots
on the idea, and help sharpen the final
recommendations made to the client. It was also
made clear to the client that this is was an
exploratory pilot project, as evidenced by the
Research Questions listed in the project proposal to
the client (Appendix 2) where the expected
outcome of the project was “A set of
recommendations on the value and use of chatbots
in HR to improve the employee experience…”.
According to Ries (2011), planning for the Build-
Measure-Learn (BML) Feedback Loop needs to be
done in reverse, i.e. “we figure out what we need to
learn … figure outwhatweneedto measure toknow
if we are gaining validated learning, and then figure
out what product we need to build to run that
experiment and get that measurement”.
As such, the subsequent sections explaining the
experiment are elaborated in reverse of the BML
Feedback Loop to demonstrate how this
experiment was planned.
Rounds of Experiments
The original goal was to build a Minimum Viable
Product (MVP) (Ries, 2011) for testing, but as this is
a very new topic and the first time the author was
exploring it, it was difficult to estimate the
timeframe needed to build the MVP and the quality
of the MVP. This made it challenging to manage
client expectations on the output to expect.
As a solution, it was then decided to set a minimum
goal of producing a low-fidelity prototype, and a
stretch goalofproducing an ERIS chatbot MVP.This
was also in line with the Lean Startup methodology
of gaining learnings and feedback as early in the
process as possible.
The client was provided with interim updates at the
end of each experiment.
Data Collection and Interviewee Selection
Qualitative Interviews
The prototype testing was conducted via qualitative
interviews for the following reasons:
1. Target audience universe is small. Headcount
for PubComms SG is approximately 250, and
therefore any quantitative research would need
relatively large numbers which was not feasible
due to the time constraints
2. Main objective is learning. The purpose of the
experiments was not just to validate or
invalidate hypotheses, it was also to gain other
insights that may help with any pivots (Ries,
2011, p. 178) which cannot be achieved with
closed-ended, quantitative research
Face-to-face, semi-structured interviews, i.e.
interviews conducted following an interview guide
using open-ended questions, were conducted
among employees.Figure 6.: Build-Measure-Learn Feedback
Loop (Ries 2011, p. 75)
Ideas
BUILD
Product
MEASURE
Data
LEARN
Minimize total time
through the loop
REDACTED

18.
18
Employees were first given the prototype to try, and
then asked for their reactions to the prototype and
thoughts on the potential usefulness and
applications of chatbots in the HR function of
PubComms SG. Responses from interviewees were
anonymised, coded and categorised for analysis.
Using qualitative interviews as a research method
would not only help to gain additional insights, but
it would also ensure that there was no confirmation
bias – defined as “seeking or interpreting evidence
in ways that are partial to existing beliefs,
expectations, or a hypothesis in hand” (Nickerson,
1998) – as the responses would be non-prompted.
As with allqualitative data analysis,there is a certain
levelofsubjectivity as interviewee responses are not
standardized, and there is a level of judgement and
interpretation required to code the data.
Interviewee Selection
A small sample of employees, 6 employees for each
round of experiment (12 individuals across two
rounds of experiments) was chosen.
To ensure a fair representation of employees and
reduce sampling bias (Uwex.edu, n.d.), the client
selected the employee interviewees for each round
based on a provided profile matrix (Table 1).
The client was also asked to choose employees from
a mix of job roles and brands. Eventually, 11 of the
12 selected interviewees responded within the
response timeframe and were interviewed.
Experiment Round 1 with Twine – New
Insights
Learning Objective: Value Over Growth
According to Ries (2011, p. 81), the riskiest elements
of any plan, are “leap-of-faith” assumptions,
because the success of the venture rests on them.
Two of the most important are the value hypothesis
and the growth hypothesis.
The value hypothesis tests whether a product or
service really delivers value to customers once they
are using it, and growth hypothesis tests how new
customers will discover a product or service (Ries
2011, p. 61).
For this experiment, the value hypothesis was
chosen over the growth hypothesis for testing as this
is the problem statement we were trying to answer,
andemployees wouldhaveno problems discovering
the chatbot as there are internal communication
tools to publicize the availability of this tool.
Going back to the problem statement, “How might
we explore the potential and applications of
chatbots in HR, specifically in PubComms SG, to
improve the employee experience?”, the
experiment sought to find interviewees’ feedback
on two aspects:
1. Potential of HR chatbots in PubComms SG to
improve the employee experience
2. Applications of HR chatbots in PubComms SG
to improve the employee experience
If these two statements yielded positive results,
then it would support a client recommendation to
invest in one.
Measure: Using Literature Review as a
Starting Point
Using the learnings from the Literature Review as a
starting point, the metrics of the experiment were
defined as non-prompted responses from
interviewees on the following:
1. Potential of HR chatbots in PubComms SG to
improve the employee experience
o 24-hour service
o Quick answers
o Getting an instant response
o Ability to self-serve without having to talk
to a customer service agent
o Get basic information on the policy

19.
19
2. Applications of HR chatbots in PubComms SG to
improve the employee experience
o Ability of interviewees to envision 2 or more
uses of chatbots in the HR function
Two assumptions were made:
• Non-prompted feedback from interviewees
would reflect their priorities and desired benefits
from chatbots
• If interviewees perceived that HR chatbots
provided value, they would be able to envision
multiple possibilities for its application
Other Measurements and Data
Other questions were also developed to mine
information that would help in exploring other areas
of opportunities (see Appendix 3 for the Qualitative
Interviewing Framework that was used as a guide to
conduct the interviews).
This included seeking information from employees
on:
• Product features anddesign thatemployees like,
that would help with subsequent iterations of
the chatbot
• Other employee needs that could be solved,
that would help the HR team to improve the
overall employee experience
• Other technology tools that employees may
desire for use in PubComms SG that may help
identify other areas of opportunity for
technology investment in the company
• Their perceptions of the role of chatbots within
the HR team, to better understand how best to
deploy technology and human resources, from
an end-user perspective
There were also other questions embedded into the
interview for the benefit of the client, particularly
two questions:
1. How would you describe the quality of your
interactions with the HR team?
2. How would you rate the HR team’s use of
technology to provide HR services to employees?
These questions had no significant impact on the
experiment, but were included to help the client
better understand employees’ context and
perspective of employees’ views of the HR team.
Build: Focusing on the Conversational Flow
Topic
The selected topic for the chatbot was the
Employee Referral Incentive Scheme (ERIS), where
employees stand to receive a monetary incentive if
they successfully refer their friends for open job
positions in Singapore. ERIS was chosen because it
is:
o A defined area of inquiry that would provide a
clear scope for the chatbot
o A relatively new policy that employees still
require assistance in understanding
o Not too simple that employees find it easier to
refer to policy documents
o Complex enough for the chatbot to provide
noticeable value-add to employees, but not
exceeding the author’s technical capabilities and
resources in creating the chatbot
o Of a low level of confidentiality that would not
impact PubComms SG should the prototype
need to be hosted on public platforms, e.g.
Facebook Messenger
Software
The low fidelity prototype was a html version
simulating a chatbot conversation, built using the
Twine program (https://twinery.org/). Originally
created to write online choose-your-own-adventure
games, it is a useful tool in designing and planning
the conversational flows in chatbots because it
allows for interactive, non-linear flows, like a
conversation, without requiring the creator to have
technical coding or programming expertise (Init.ai
Decoded, 2016; WillowTree, Inc., 2016; Winstead,
2016; Vaneseltine, 2014).
Twine was chosen as it provides an overview of the
conversation flow (Figure 7), to allow the author to
evaluate the effectiveness of the conversation, e.g.
help users to get to the end point as fast as possible.
Refer to Appendix 4 to access and view the ERIS
conversational flow written on Twine (requires
download of the free program to view). Video
demonstrations of the front-end and back-end view
are also included in the same appendix and shown in
Figure 8 and 9.

20.
20
Figure 7.: ERIS – Twine Conversational Flow
REDACTED

21.
21
Insights surfaced in the literature review section on
good chatbot UX were incorporated. This includes
ensuring that the prototype:
• Set the right expectations
• Allow flexibility of exploration
• Create the tone to reflect the brand personality
• Help users get help
However, as Twine does not have certain
functionalities of chatbots, the following best
practices were not incorporated:
• Use buttons to supplement free text input
• Layer AI to interpret user’s requests
• Use images or graphics to supplement text
• Introduce product features gradually
• Provide a null response
• Integration with existing business systems
• Securing information security and privacy of
information
This prototype was shown to interviewees on a
desktop browser, rather than through a messaging
platform.
Results: New Insights
Potential of HR Chatbots in PubComms SG
Interview results show that the hypothesised
benefits of chatbots to improve the employee
experience were validated.
Interviewees said that they thought that the
chatbot provided good information on the policy
and was personable. They also said that a chatbot
would provide instant answers and enable
employees to find information independently.
Employees also rated the usefulness of a chatbot in
helping to improve the employee experience 7.6 out
of a possible 10, where 10 is the most useful, and 1
the least useful.
Some interesting insights emerged from the
interviews:
Chatbot is easy to use
ERIS is a relatively more complex HR policy within
PubComms SG due to differing incentive amounts
(depends on the type of job position employees are
making referrals for), and incentive payment
processes (depends on the nature of the
employment contract of the referring employee).
The conversational flow in the prototype was
designed based on conditional logic, i.e. users were
taken through a series of questions, and based on
their answers, the chatbot would provide
information that was specificto their situation (refer
to Appendix 4 for documentation on the Twine
program).
Therefore, this feedback from interviewees,
especially when it was the most mentioned
feedback, is surprising. This indicates that a chatbot
has the potential for simplifying complex
information for users, if planned well.
Chatbots help employees to avoid embarrassment
An unexpected feedback from employees was that
chatbots would help employees to avoid
embarrassment when they needed to as HR
questions that they felt were trivial.
Chatbots help to provide anonymity
This was raised specific to situations when
colleagues are trying to understand policies on
sensitive issues: “In an Asian culture, people are
more non-confrontational, and may not want to
raise alarm bells until they are sure. Making such
information readily available without having to ask
a HR team member [for access] allows them to do
their initial research”.
“In an Asian culture, people are more non-confrontational, and
may not want to raise alarm bells until they are sure. Making such
information readily available without having to ask a HR team
member [f0r access] allows them to do their initial research.”
REDACTED

22.
22
Figure 8: ERIS – Demonstration of Twine Front End User View
VIDEO REDACTED

23.
23
Figure 9: ERIS – Demonstration of Twine Back End View
VIDEO REDACTED

24.
24
Chatbots help to relieve the HR team
Beyond seeing the benefits for themselves, most
saw the benefit of using chatbots to help relieve the
HR team’s workload so that they can redirect their
time to focus on more important issues.
Applications of HR Chatbots in PubComms SG
All interviewees mentioned at least two other areas
of applications, including:
• Company policies
• Company processes
• Company information
• Employee contract details
• New employee induction / orientation
• Training
• IT requests
Interestingly, the most of them related to
information services. Refer to Appendix 5 for the full
results and analysis of the interviews.
UX Feedback
Some areas of improvement that interviewees
suggested to improve the user experience:
• Reduce the wordiness of the chatbot
• Have a “home” button (there was a “back”
option in the prototype, but not a “home”
option)
Role of Chatbots vs HR team
Interviewees were also clear on the role of the
chatbot within the HR team. They would use a
chatbot for functional or transactional queries
and/or generic FAQs, and approach a HR member
when they need counsel on non-routine requests,
emotional issues, or exceptions to the regular
policies and processes. This seems to indicate that
chatbots can help transition HR away from a
transaction perception in employees towards an
expertise role.
Experiment Round 2 with Chatfuel –
Validation and More
Learn: Would Feedback be the Same?
Based on the learnings from Experiment Round 1,
the following decisions were made on the desired
learning from Round 2:
• Potential of HR Chatbots in PubComms SG
- To reconfirm the benefits even after the
prototype is changed from a html, desktop
version to a chatbot MVP hosted on a mobile
messaging platform
- Continue to listen and monitor if the new
insights would continue to be mentioned in
Round 2, thereby validating them
• Applications of HR Chatbots in PubComms SG
- Reconfirm the desired applications of
chatbots in HR
Build: Creating a chatbot MVP
The aim of the Round 2 Experiment was to be able
to test an ERIS chatbot MVP. The MVP was written
using Chatfuel (http://chatfuel.com/), a chatbot
builder.
Two other bot builders were considered – Botsify
and Wit.ai – but Chatfuel was eventually chosen as
it had the best combination of elements that met
the design considerations and required features of
the experiment (Table 2).

25.
25
Figure 10: ERIS Chatbot Demonstration on Facebook Messenger
VIDEO REDACTED

26.
26
Figure 11: ERIS Chatbot – Chatfuel Demonstration
VIDEO REDACTED

27.
27
A key factor in deciding which chatbot builder to use
was the messaging platform through which users
could access and interact with the chatbot. Offering
a chatbot through an owned PubComms SG asset
was not possible as PubComms does not have any
SG-specific intranet or website.
A quick review of the chatbot builders available in
the market show that most of them design their
services to be hosted on a few key messaging
platforms (Levinson, 2016; BotStory, 2016):
Facebook Messenger, Slack, Skype, Telegram and
Kik. As Facebook Messenger was the platformlikely
to be most used among PubComms SG employees
on their mobile phones, it was one of the criteria
used to narrow the list of chatbot builders.
Given the above, an additional question was added
to the interview guide on interviewees’ thoughts on
accessing a HR chatbot via Facebook Messenger.
In adapting the Twine prototype to Chatfuel, these
additional features were added to the MVP:
• Include a ‘home’ button or its equivalent
• Provide a null response
• Allow users to type in their own queries / open-
ended questions
• More efficient user / conversational flows, i.e. less
steps to get to the required information
• Reduce wordiness and use more graphics
• Layer AI to interpret users’ requests
• Introduce product features gradually (Dropbox)
Features recommended in the literature review that
were not included:
• Integration with existing business systems
• Securing information security and privacy of
information
This was not required as the MVP was designed to
be a standalone prototype (so it could be built
quickly) and thus not needing additional
information security measures.
Appendix 6 includes source files for the two video
demonstrations of the ERIS chatbot MVP you see in
Figure 10 and 11. One shows the user view when
interacting with the ERIS chatbot via Facebook
Messenger, and the other is a demonstration of the
architecture of the ERIS chatbot on Chatfuel.
*To access the ERIS chatbot MVP on Chatfuel
(backend view), please email the author at
amy.kong@hyperisland.sg for an access link (valid
for 24 hours). To interact with the ERIS chatbot,
please use the Facebook Messenger app and search
for @PubCommsERIS (case-sensitive) or scan the
Messenger Code (Figure 12.)
Results: Validation and More
Interview results continued to validate the
hypothesised benefits of chatbots to improve the
employee experience were validated.
In fact, the usefulness rating of chatbots in
improving the employee experience rose from an
average of 7.6 in Round 1 to 7.9 in Round 2.
As this group of interviewees did not interact with
the Twine prototype, there was no opportunity to
investigate if this increase in rating was due to the
switch in interaction from a desktop web browser to
Facebook Messenger.
Figure 12: ERIS Chatbot Messenger Code
To interact with ERIS:
1. Launch the Facebook Messenger App
2. From Home, tap the icon
3. Tap your picture at the top of the page
4. Tap Scan Code
5. Scan the image on the left
REDACTED
REDACTED

28.
28
Interviewees (except for one) were also able to
envision at least two other areas of applications of
chatbots in HR, and were still able to differentiate
when they would use a chatbot and when to
approach a HR team member.
A more nuanced view emerged on the feedback on
avoiding embarrassment. Interviewees said that
chatbots wouldhelp when employeesneededto ask:
o Trivial questions (same feedback in Round 1)
o Questions the employee is supposedto know the
answers to
o Questions the employee perceives may impact
their image/reputation among co-workers
Three additional areas of applications of chatbots in
HR raised by interviewees in Round 2 include:
• Making company announcements
• Collecting employee suggestions
• FAQ facility for hiring managers, e.g. if the
company is still within the ‘S pass’ (a type of
employment pass in Singapore) government
quota for them to be able to hire foreigners
The wordiness of the chatbot continued to be a
consistent feedback for improvement in Round 2.
Balancing between wordiness and omission of
critical information will need to be considered,
particularly in HR, where there could be legal or
employment implications if important information
is not communicated to the employee.
Interviewees agreed that a HR chatbot needs to be
hosted outside of Facebook, as it is a personal social
media platform and HR chatbots are for
professional purposes.
One employee pointed out that the chatbot should
be hosted on a platform that most employees use,
and another pointed out that most employees use
mobile phones for work, which makes a messenger-
based chatbot appropriate.
Overall, interviewees in Round 2 seemed to show
more enthusiasm over the prototype. There were
more expressions of excitement over the ERIS
chatbot MVP. Some quotes from the interviewees:
“Good that we are using technology in HR. We do it in
our business. The fact that we are taking HR online
makes it exciting. Now that I know that PubComms SG
is considering it, I really want it and am excited for it. I
can’t see how it would be a disadvantage.”
“This is good. It primes employees for more future
forward communications. That’s how it will be in the
next few years. It will help people to learn how to apply
it in their own (client) brands.”
“We need to do things faster now, and something like
this helps to get things done faster, especially when
teams are so lean now.”
“It is overwhelming how smart it is…I like the language
used. It is friendly and approachable, and it makes me
feel like I can ask any question.”
“We don’t have to wait for a response from HR or be
dependent on the HR person being around.
Currentlywe pass a lot ofmessages around… so that
HR doesn’t receive repeated questions. This allows
you to find the answer directly yourself.”
Overall Findings and Conclusion
Problem Statement Exploration and
Validation
After two rounds of experiments, results indicate
that employees think that chatbots have the
“This is good. It primes employees for more future forward
communications. That’s how it will be in the next few years. It will
help people to learn how to apply it in their own (client) brands.”

29.
29
potential to improve the employee experience, and
can be applied in various areas in HR within
PubComms SG.
Specifically, the benefits to employees include:
• Instant access and answers
• Ability to find information independently
• Provide good information
• Personable
Interviewees found chatbots useful, and were able
to envision how chatbots could be used in the HR
function, including in:
• Information services: company information,
policies, processes, and status updates
• Training
• Information collection and dissemination:
broadcasting company announcements and
collecting employee suggestions and feedback
Interviewees even suggested that chatbots could be
used to log IT service requests.
Finally, PubComms SG HR chatbots should be
hosted on appropriate messaging platforms
suitable for work use.
Additional Insights
• Chatbots seem to be able to play a role in
simplifying complex information for employees,
when the conversational flow of the chatbot is
planned well
• Chatbots help employees to avoid feeling
embarrassed when asking certain types of
questions
• Chatbots can provide employees with a sense of
anonymity on sensitive issues
• In the longer term, chatbots could provide the
opportunity to shift perceptions of the HR team
from an administrative / transactional role to
more of an expert role
Limitations of the Experiment
Small Sample
As the experiment was designed to generate
qualitative feedback, it also meant that the sample
size is small and may not be representative of
PubComms SG, although some of the feedback was
validated in Round 2 of the experiment.
Lack of Quantitative Metrics
As data collected was qualitative, there were no
quantitative metrics that were collected. One
quantitative metric that could provide valuable
insight (and justification to management) if future
experiments are conducted, is the time saved in
using a chatbot to complete HR tasks, versus the
current approach.
Confined Area of Enquiry
The chatbot was built based on ERIS, which is a very
specific HR policy within PubComms SG. In
implementing any HR chatbot, the areas of enquiry
would need to be broader than one specific policy,
and the chatbot was not tested on its ability to filter
or direct questions down different topic paths. This
may impact the “ease of use” feedback that was
seen in this experiment.
“We need to do things faster
now, and something like this
helps to get things done
faster, especially when teams
are so lean now.”
“Good that we are using
technology in HR. We do it in
our business. The fact that we
are taking HR online makes it
exciting. Now that I know
that PubComms SG is
considering it, I really want it
and am excited for it. I can’t
see how it would be a
disadvantage.”

30.
30
Review and Recommendations
Yay or Nay?
Based on the Literature Review and Experiment,
there is significant potentialandvalue in investing in
HR chatbots to improve the employee experience
and improve employee productivity by reducing
time spent on completing HR tasks.
The availability of chatbot builders and vendor
solutions in the market now also lower the barriers
of entry for any investment in building chatbots.
How Much Does It Cost?
It depends on the complexity of the chatbot. There
are also still no industry standards for pricing, and
various vendors and platforms have different
business models which make it hard to do a cross-
vendor or solution comparison of costs.
However, to help with understanding the cost
implications, Table 3 provides a list of budget items
to be considered.
It Could Save SGD 125,000 Annually
Based on conservative estimates (see Table 4), a
chatbot could save the company SGD 125,000
annually, excluding the cost of building the chatbot.
This is a conservative estimate as a chatbot’s instant
response capability also allows employees to reduce
multi-tasking as a result of waiting for responses
from the HR team. According to the American
Psychological Association (2006), “even brief
mental blocks created by shifting between tasks can
cost as much as 40 percent of someone's productive
time.”
If you add in task automation capabilities into the
chatbot, cost saved would also increase.
Suggested Timeline and Next Steps
It may be more appropriate to look at a timeline of 6
months to launch an “official” HR chatbot for the
following reasons:
Allow Time for Market to Mature
The market is still at a relatively early stage of
development.
This should settle down somewhat in 6 months for
PubComms SG to get a clearer perspective of the
real possibilities of chatbots.
Allow Time for More Vendor Solutions Emerge
and Stabilize
PubComms SG should also consider off-the shelf
solutions, but many of these solutions are US-based,
or still in beta versions. Waiting 6 months will allow
more vendor solutions to emerge and become more
stable.
Also, given the huge messaging usage in Asia, Asia-
specific or global solutions should emerge in the
market soon.
Other Decisions Around the Implementation of
the Chatbot
There are some areas that need to be reviewed and
decided, which will influence how the HR chatbot in
PubComms SG will be implemented. This includes:

31.
31
Topic/s of the Chatbot
While the prototypes were built based on ERIS for
specific reasons, ERIS is still a policy that is relevant
only to a subset of employees, used infrequently. It
may be worth considering a different set of policies
to build the chatbot around, for example, top 10
FAQs, leave entitlements, medical benefits, list of
panel of doctors etc.
Choice of Messaging Platform
Due to the confidentiality of HR policies and
information, an appropriate messaging platform
needs to be selected. At the same time, the choice
of platform needs to be easily accessible by
employees via their personal mobile devices. Some
options that can be considered:
• Skype for Business (Lync), the official corporate
messaging tool for PubComms SG. This assumes
that this feature will be made available to all
employees on their mobile phones
• Create a mobile-friendly intranet to house
company information common to all brands, and
also use it to host a chatbot
Integration with Publicis Groupe Enterprise HR
Software, Data Protection and Security
A decision needs to be made on how personalised
the chatbot should be. Personalising interactions
requires integrating the chatbot with PubComm
SG’s existing enterprise HR software to access
employee data, and as such issues of data
protection and security need to be addressed.
On the upside, SAP, the enterprise HR software
vendor used by Publicis Groupe, is partnering Kore
to integrate chatbot products into its software
(Besser, 2016), which resolves the security issues.
On the downside, the speed at which chatbots can
be deployed in PubComms SG may be slowed, as
any use of the enterprise HR software will need to
be authorised by the Pubicis Groupe global team,
and vetted and handled by ReSources IT (Publicis
Groupe IT department).
Analytics
There is an opportunity for PubComms SG to collect
data to monitor and improve the performance of
the chatbot, understand employees concerns better
and eventually shape the Talent strategy of the
company. The type of data to be collected should be
considered during the design of the chatbot. This
was not a focus for the experiment as the sample
size was too small to collect any meaningful data.
Employee Privacy and Anonymity
Given that some employees have said that they feel
that chatbots can provide anonymity, there needs
to be discussion on how much,andwhat data should
be collected, and what level of anonymity will be
provided.
Accuracy and Completeness of Information
Scenarios where information provided could have a
financial impact to the employee or company, or if
an employee’s critical employment decision is
dependent on a complete understanding of a
specific HR policy, need to be discussed to ensure
that proper check mechanisms are put in place. No
doubt, these situations happens now without the
use of chatbots, but they need to be discussed and
the handling approach agreed on to ensure that
there is a proper resolution mechanism in place.
REDACTED
REDACTED
REDACTED

32.
32
HR Talent
HR employees with a new skill set may be needed to
drive a technology project such as this. The selected
employee may not need to be a technical specialist,
but needs to be able to understand the project
parameters and work with vendors. In the longer
term, a review will be needed of the composition
andskillsetoftheHRteamasmoretechnologytools
are used in the HR function.
Other Recommendations and
Considerations
While there are many Do-It-Yourself chatbot
builders in the market, it may be best for
PubComms SG to hire a vendor to develop one if a
customised solution is desired. Integrating the
chatbot with the enterprise HR software will need
considerable technical skill. Additionally, new
chatbot features are constantly being added, and
unless an employee is dedicatedto focus on building
the chatbot, it would be difficult to keep up with the
constant upgraded capabilities of the chatbot
builders.
The Time is Now
Even as the suggested timeline is in 6 months’ time,
the preparatory work needs to start now, as some of
the considerations listed will involve careful
consideration and multiple stakeholders that need
time to resolve. MVPs can still be tested with a
curated employee group of early adopters so that
learnings are accumulated and can be baked into
the “official” chatbot when ready.
Acknowledgements
My journey at Hyper Island would not have been possible without the advice and constant moral
support from friends and family. I would like to specially thank the following who have
contributed directly to my AWBP journey:
Dan Spencer
Chief Talent Officer, Publicis Communications, Singapore and Australia / New Zealand
Leong Yeng Wai
Student, Hyper Island Singapore
Benjamin Koe
Product Lead, Dentsu Aegis Global Data Innovation Centre
Narendra Nag
Chief Digital Officer, Xynteo
Lim Chwen Yiing and Lim Han Boon
Best Buddies
Jonathan Briggs
Co-Founder and Singapore Academic Director, Hyper Island
Jarrod Howe
Operations Director, Hyper Island Singapore

33.
33
Reflections
This journey leading up to the completion of this Advanced Work Based
Project (AWBP) has been quite an interesting and enlightening one. These
are some of the learnings and experiences that have shaped this AWBP:
Complexity of Chatbots and The Landscape
Much of the literature talks about how easy it is to create chatbots. It is
true, but it is also false. It is true that many of the tools now in the market,
particularly the chatbot builders, are relatively easy to use. However,
before getting to the pointofbuilding a chatbot,company decision makers
need to first understand if they are making the right choice in investments,
which means understanding the landscape, and that is where it gets
complex.
When I first started this project, I had no idea where to start, as I have no
coding or IT specialist experience and my experience with technology is
purely from an end-user perspective. As I started the research, it seemed
like falling down a never-ending rabbit hole. Just to understand chatbot
literature that is regularly peppered with jargon, you must know how it fits
vis-à-vis AI, machine learning, NLP, NLU, Natural Language Generation.
What’s the difference between a chatbot and a conversational agent?
What is the difference between a conversational agent and a dialog
system?
That’s just understanding chatbots andits possible functions.To create the
chatbot, you need to understand the different approaches to building one,
then understand the comparative differences across the plethora of tools
to choose one to use. And the list goes on. I first worked on wit.ai to start
building the chatbot as recommended by chatbot-related articles as being
easy to use. But it turned out to be an easy tool for developers to use, and
not for non-coders like me. So, I had to abandon the work I started in wit.ai
and re-start the search for a chatbot builder.
For a HR teamthat may not have a large IT departmentto rely on for advice,
or even a for CEO who is interested in investing in one, the landscape can
be hard to decipher.
Jargon Everywhere
The proliferate use of jargon is only part of the problem. The other part of
the problem is that at this stage of market development, terminology is not
clearly defined or differentiated, or writers have a poor understanding of
chatbot technology. Sometimes the same terms mean different things, or
different terms mean the same thing. Some articles refer to AI as different
from machine learning. Some articles talk about chatbot services while
others talk about chatbot platforms when they mean the same thing.
As such, I tried to be as clear as possible in this report on the different terms
and definitions, and explain the chatbot landscape as simply as I could,
based on my understanding. As this report is meant to be shared with
PubComms SG, hopefully this will help provide clarity for the client.
Easy for Whom?
It is easy to use a chatbot builder, but it may not be simple to build a
chatbot, or at least build a good one. As I was using wit.ai, I was not able to
understand how the conversation with the user would flow (that would help
me evaluate its effectiveness), as the tool does not help you to do so. This
is the same with other chatbot builders that I explored. Adding on to that
stress was the lack of time to conduct the experiment,which was how I then
researched and found Twine. Even as I was using Twine, I realised that you
need an understanding of conditional logic. Upon consulting my ex-
colleague who has developer experience, I also realised that you need to
plan the conversational flow not in sequential steps, but in blocks of
information, and then create paths or flows to the informational blocks.
REDACTED

34.
34
Reflections (cont’d)
In short, anybody can build a chatbot, if that anybody understands how to
plan conversational flows, understands conditional logic etc. Along the
way, you’ll encounter other terms like ‘rails’, ‘conversational trees’, ‘dialog
trees’ and so on, and down the rabbit hole you go again. Nevertheless, it
is true that the barrier to entry for building chatbots are now far lower
than before.
Drinking From a Firehose
Chatbots are such a hot topic now, that news articles and blog posts are
generated almost every day. I found it difficult to keep track of the
emerging articles and evaluate them for inclusion in this paper. It was also
a challenge to select a balance of commercial vs academic sources.
Firstly, academic journals do not necessarily reflect the current state of
business discussion around the topic, and secondly, they tend to be very
technical. There was significant temptation to use data primarily from
commercial articles as they provide quick and short summaries. I have
tried to refer to academic sources in this paper for key definitions, terms,
concepts and key milestones and discussions, and kept the references to
commercial articles confined to areas where the academic journals may
not touch upon.
Iteration vs Documentation
When using Chatfuel, changes to any part of the chatbot is reflected ‘live’.
There is also no option for version downloads or control. As such, during
the experimentation phase, I found it difficult to keep track of the
changes that I made to the chatbot. In the longer term, it also raises
questions on how to restore chatbots should there be any changes that
need to be reversed.
When is Customer Feedback Valid?
Part of the iteration process is seeking qualitative feedback from
customers. One of the questions that I kept asking myself was, “at what
point is customer feedback considered valid or an outlier?”. Ultimately, I
made the decision based on what I felt made sense and was in line with
the overall objective of the chatbot, but it seemed very subjective to me. If
given the opportunity, I would ask current Lean Startup practitioners how
they make such decisions, particularly with qualitative testing with small
sample sizes
Challenge in Integrating with Enterprise HR Software
I anticipate that there will be challenges in getting this chatbot project
running in PubComms SG if backend integration with the enterprise HR
software is needed. Such requests will need to be approved by the Global
Publicis Groupe team, and actioned by ReSources IT, which is the internal
IT shared services function. This affects the speed at which backend
functions in PubComms SG can capitalise on emerging opportunities.
PubComms SG may be better off creating a generic chatbot now, then
present the case study to the Global team for endorsement, to increase
the speed of implementation.
REDACTED

35.
35
References
Ady, M. (2016). How to Build a Chatbot Your Users Will Love. [online] VentureBeat. Available at: http://venturebeat.com/2016/08/17/how-to-build-a-chatbot-your-users-will-love/
[Accessed 3 Jan. 2017].
American Psychological Association. (2006). Multitasking: Switching costs. [online] Available at: http://www.apa.org/research/action/multitask.aspx [Accessed 5 Jan. 2017].
Aspect, (2016). 2016 Aspect Consumer Experience Index. [online] Phoenix. Available at: https://www.aspect.com/globalassets/2016-aspect-consumer-experience-index-
survey_index-results-final.pdf [Accessed 1 Jan. 2017].
Aspect.com. (n.d.). Aspect Mila: The Agent’s Personal Assistant. [online] Available at: https://www.aspect.com/solutions/workforce-optimization/aspect-mila [Accessed 2 Jan.
2017].
Ball, P. (2015). The truth about the Turing Test. [online] Bbc.com. Available at: http://www.bbc.com/future/story/20150724-the-problem-with-the-turing-test [Accessed 31 Dec.
2016].
Batalion, A. (2016). “Bot” is the Wrong Name.. and Why People Who Think They are Silly are Wrong. [online] Medium. Available at: https://medium.com/lightspeed-venture-
partners/bot-is-the-wrong-name-and-why-people-who-think-they-are-silly-are-wrong-dc0c0b76ae18#.y4x0znitc [Accessed 1 Jan. 2017].
Bayerque, N. (2016). A Short History of Chatbots and Artificial Intelligence. [online] VentureBeat. Available at: http://venturebeat.com/2016/08/15/a-short-history-of-chatbots-and-
artificial-intelligence/ [Accessed 1 Jan. 2017].
Besser, L. (2016). Kore bots for SAP empower every employee to delegate like a boss. [online] Ideas.sap.com. Available at: https://ideas.sap.com/D32372 [Accessed 4 Jan. 2017].
Bhaduri, A. (2016). The Digital Tsunami: HR. [online] Abhijit Bhaduri's Official Website. Available at: http://www.abhijitbhaduri.com/index.php/2016/08/the-digital-tsunami-hr/
[Accessed 2 Jan. 2017].
Blank, S. (2013). Why the Lean Start-Up Changes Everything. [online] Harvard Business Review. Available at: https://hbr.org/2013/05/why-the-lean-start-up-changes-everything
[Accessed 22 Dec. 2016].
Blank, S. (2015). Why Build, Measure, Learn – isn’t just throwing things against the wall to see if they work – the Minimal Viable Product. [online] Steve Blank. Available at:
https://steveblank.com/2015/05/06/build-measure-learn-throw-things-against-the-wall-and-see-if-they-work/ [Accessed 25 Dec. 2016].
BotStory. (2016). Top messaging platforms supporting bots. [online] Available at: http://botstory.co/top-platforms-support-bots-chatbots/ [Accessed 28 Dec. 2016].
Boulton, C. (2016). From tacos to HR, chatbots make it personal. [online] CIO. Available at: http://www.cio.com/article/3063051/instant-messaging/from-tacos-to-hr-chatbots-
make-it-personal.html [Accessed 2 Jan. 2017].
Business Insider. (2016). Messaging Apps are Now Bigger than Social Networks. [online] Available at: http://www.businessinsider.com/the-messaging-app-report-2015-
11?IR=T&r=US&IR=T [Accessed 1 Jan. 2017].

36.
36
Chowdhury, G. (2005). Natural language processing. Annual Review of Information Science and Technology, 37(1), pp.51-89.
Christensen, C., Hall, T., Dillon, K. and Duncan, D. (2016). Know Your Customers’ “Jobs to Be Done”. [online] Harvard Business Review. Available at: https://hbr.org/2016/09/know-
your-customers-jobs-to-be-done [Accessed 26 Dec. 2016].
Dan, A. (2016). Consultants Are Eating The Agencies' Three-Martini Lunch. [online] Forbes.com. Available at: http://www.forbes.com/sites/avidan/2016/04/25/consultants-are-
eating-the-agencies-three-martini-lunch/#5ba472b63aba [Accessed 30 Dec. 2016].
Deutscher, M. (2016). FirstJob’s Mya is the latest chatbot that aims to automate recruiting. [online] SiliconANGLE. Available at: http://siliconangle.com/blog/2016/07/11/firstjobs-
mya-is-the-latest-chatbot-that-aims-to-automate-recruiting/ [Accessed 2 Jan. 2017].
Dishman, L. (2016). This Chatbot Can Make Sure Your Resume Won't End Up In A Black Hole. [online] Fast Company. Available at: https://www.fastcompany.com/3061677/the-
future-of-work/the-chatbot-who-can-make-sure-youll-never-get-radio-silence-after-applyin [Accessed 2 Jan. 2017].
Dunn, J. (2016). We put Siri, Alexa, Google Assistant, and Cortana through a marathon of tests to see who’s winning the virtual assistant race — here’s what we found - Business
Insider. [online] Business Insider. Available at: http://www.businessinsider.sg/siri-vs-google-assistant-cortana-alexa-2016-11/?r=US&IR=T#rfBHqO3D8eSOZMdp.97
[Accessed 2 Jan. 2017].
Elliott, S. (2002). Advertising's Big Four: It's Their World Now. [online] Nytimes.com. Available at: http://www.nytimes.com/2002/03/31/business/advertising-s-big-four-it-s-their-
world-now.html?pagewanted=all&src=pm [Accessed 20 Dec. 2016].
Ember, S. (2016). Ad Agencies Need Young Talent. Cue the Beanbag Chairs.. [online] Nytimes.com. Available at: http://www.nytimes.com/2016/04/19/business/media/ad-agencies-
need-young-talent-cue-the-bean-bag-chairs.html?_r=0 [Accessed 31 Dec. 2016].
Epstein, R., Roberts, G. and Beber, G. (2008). Parsing the turing test. 1st ed. New York: Springer, pp.181-210.
Farmer, M. (2015). Madison Avenue Manslaughter. 1st ed. New York: LID Publishing Ltd.
Fromowitz, M. (2016). Commoditization: The biggest threat facing ad agencies today. [online] Campaign Asia. Available at: http://www.campaignasia.com/article/commoditization-
the-biggest-threat-facing-ad-agencies-today/425997 [Accessed 30 Dec. 2016].
Futurism. (n.d.). The History of Chatbots [INFOGRAPHIC]. [online] Available at: http://futurism.com/images/the-history-of-chatbots-infographic/ [Accessed 1 Jan. 2017].
Gartner.com. (2015). Gartner Says Weak Mobile Customer Service Is Harming Customer Engagement. [online] Available at: http://www.gartner.com/newsroom/id/2956618
[Accessed 1 Jan. 2017].
Grayson, A. and North, G. (2016). How the Advertising Industry is Wasting Talent and What We Can Do About It. [online] Medium. Available at:
https://medium.com/@andygrayson/how-the-advertising-industry-is-wasting-talent-and-what-we-can-do-about-it-4824411b9006#.8h9lmf6e3 [Accessed 31 Dec. 2016].
Greenfield, R. (2016). Chatbots Are Your Newest, Dumbest Co-Workers. [online] Bloomberg.com. Available at: https://www.bloomberg.com/news/articles/2016-05-05/chatbots-are-
your-newest-dumbest-co-workers [Accessed 2 Jan. 2017].

37.
37
Griffin, A. (2014). Computer becomes first to pass Turing Test in artificial intelligence. [online] The Independent. Available at: http://www.independent.co.uk/life-style/gadgets-and-
tech/computer-becomes-first-to-pass-turing-test-in-artificial-intelligence-milestone-but-academics-warn-9508370.html [Accessed 31 Dec. 2016].
Hobson, N. (2016). Are chatbots in the workplace the entry point to cognitive personal assistants? | NevilleHobson.com. [online] NevilleHobson.com. Available at:
http://www.nevillehobson.com/2016/05/10/chatbots-workplace-cognitive/ [Accessed 1 Jan. 2017].
Howard, T. (2016). The Guide To Designing A Magical Chatbot Experience. [online] Chatbots Magazine. Available at: https://chatbotsmagazine.com/the-guide-to-designing-a-
magical-chatbot-experience-part-1-efbf32444448#.wyex0oqer [Accessed 3 Jan. 2017].
Init.ai Decoded. (2016). The Product Designer’s Guide to Conversational Commerce. [online] Available at: https://blog.init.ai/the-product-designers-guide-to-conversational-
commerce-cbe466753add#.gn5sjy8bk [Accessed 27 Dec. 2016].
Jain, A. (2016). The Flow Framework — How to build a kickass UX for your Chat-bot? (Part — II). [online] ChatterOn. Available at: https://blog.chatteron.io/how-to-build-a-kickass-ux-
for-your-chat-bot-part-ii-the-flow-framework-811355905249#.6dlq6d3qw [Accessed 3 Jan. 2017].
Johnson, B. (2015). State of the Agency Market: What You Need to Know. [online] Adage.com. Available at: http://adage.com/article/digital/datacenter-agency-report-2015-
charts/298214/ [Accessed 30 Dec. 2016].
Knowledge@Wharton. (2016). The Rise of the Chatbots: Is It Time to Embrace Them? - Knowledge@Wharton. [online] Available at:
http://knowledge.wharton.upenn.edu/article/rise-chatbots-time-embrace/ [Accessed 31 Dec. 2016].
Kojouharov, S. (2016). Ultimate Guide to Leveraging NLP & Machine Learning for your Chatbot. [online] Chatbot’s Life. Available at: https://chatbotslife.com/ultimate-guide-to-
leveraging-nlp-machine-learning-for-you-chatbot-531ff2dd870c#.zaqk0o35t [Accessed 2 Jan. 2017].
Lamprecht, E. (2016). The Difference Between UX and UI Design-A Layman’s Guide. [online] Blog.careerfoundry.com. Available at: http://blog.careerfoundry.com/ui-design/the-
difference-between-ux-and-ui-design-a-laymans-guide/ [Accessed 4 Jan. 2017].
Levinson, R. (2016). The tools every bot creator must know. [online] Chatbots Magazine. Available at: https://chatbotsmagazine.com/the-tools-every-bot-creator-must-know-
c0e9dd685094#.v6h0lbonf [Accessed 28 Dec. 2016].
Lo, M. (2014). UI, UX: Who Does What? A Designer's Guide To The Tech Industry. [online] Co.Design. Available at: https://www.fastcodesign.com/3032719/ui-ux-who-does-what-a-
designers-guide-to-the-tech-industry [Accessed 4 Jan. 2017].
Loebner, H. (n.d.a). In Response. [online] Loebner.net. Available at: http://loebner.net/Prizef/In-response.html [Accessed 31 Dec. 2016].
Loebner.net. (n.d.b). Home Page of the Loebner Prize in Artificial Intelligence. [online] Available at: http://www.loebner.net/Prizef/loebner-prize.html [Accessed 31 Dec. 2016].
Maass, D. (2014). Answers and Questions About Military, Law Enforcement, and Intelligence Agency Chatbots. [online] Electronic Frontier Foundation. Available at:
https://www.eff.org/deeplinks/2014/04/answers-questions-about-military-law-enforcement-and-intelligence-agency-chatbots [Accessed 2 Jan. 2017].
Marcus, D. (2016). Messenger Platform at F8. [online] Newsroom.fb.com. Available at: http://newsroom.fb.com/news/2016/04/messenger-platform-at-f8/ [Accessed 1 Jan. 2017].

38.
38
Marr, B. (2016). What Is The Difference Between Artificial Intelligence And Machine Learning?. [online] Forbes.com. Available at:
http://www.forbes.com/sites/bernardmarr/2016/12/06/what-is-the-difference-between-artificial-intelligence-and-machine-learning/3/#657340f11a30 [Accessed 1 Jan. 2017].
Marshall, J. and Alpert, L. (2016). Publishers Take On Ad-Agency Roles With Branded Content. [online] WSJ. Available at: http://www.wsj.com/articles/publishers-take-on-ad-
agency-roles-with-branded-content-1481457605 [Accessed 30 Dec. 2016].
Maurya, A. (2012). Running Lean: Iterate from a Plan A to a Plan that Works. 2nd ed. Sebastopol, CA: O'Reilly.
McNeale, M. and Newyear, D. (2013). Introducing Chatbots in Libraries. Library Technology Reports, 49(8), pp.5 - 10.
Medium. (2016). 聊天機器人市場版圖 — Chatbots Landscape. [online] Available at:
https://medium.com/@compose.ai/%E8%81%8A%E5%A4%A9%E6%A9%9F%E5%99%A8%E4%BA%BA%E5%B8%82%E5%A0%B4%E7%89%88%E5%9C%96-chatbots-
landscape-f89302206875#.wk2de23dy [Accessed 2 Jan. 2017].
Meeker, M. (2016). 2016 Internet Trends Report. [online] Kpcb.com. Available at: http://www.kpcb.com/internet-trends [Accessed 1 Jan. 2017].
Mindshare, Goldsmiths University of London, (2016). Humanity in the Machine. [online] Available at:
http://www.mindshareworld.com/sites/default/files/MINDSHARE_HUDDLE_HUMANITY_MACHINE_2016_0.pdf [Accessed 1 Jan. 2017].
Morrison, M. (2016). Sprint Names Exec Creative Director to Run New In-House Agency. [online] Adage.com. Available at: http://adage.com/article/agency-news/sprint-staffs-house-
agency/304045/ [Accessed 30 Dec. 2016].
Mumbrella Asia. (2016). Lou Dela Pena handed broader role leading Publicis Communications in Singapore - Mumbrella Asia. [online] Available at:
http://www.mumbrella.asia/2016/01/lou-dela-pena-handed-broader-role-leading-publicis-communications-in-singapore/ [Accessed 20 Dec. 2016].
myclever Agency, (2016). Chatbots: A Consumer Research Study. [online] London. Available at:
http://www.slideshare.net/mycleveragency?utm_campaign=profiletracking&utm_medium=sssite&utm_source=ssslideview [Accessed 1 Jan. 2017].
Newman, J. (2016). How The New, Improved Chatbots Rewrite 50 Years Of Bot History. [online] Fast Company. Available at: https://www.fastcompany.com/3059439/why-the-new-
chatbot-invasion-is-so-different-from-its-predecessors [Accessed 31 Dec. 2016].
Newton, C. (2016). Can AI fix education? We asked Bill Gates. [online] The Verge. Available at: http://www.theverge.com/2016/4/25/11492102/bill-gates-interview-education-
software-artificial-intelligence [Accessed 2 Jan. 2017].
Nextit.com. (n.d.). SGT STAR helps potential recruits learn about Army life. [online] Available at: http://www.nextit.com/work/army.php [Accessed 2 Jan. 2017].
Nickerson, R. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), pp.175-220.
Oetting, J. (2015). The Biggest Threats to the Agency Model. [online] Blog.hubspot.com. Available at: https://blog.hubspot.com/agency/threats-agency-
model#sm.000q8a2qc1192eo6ypt159w7mslve [Accessed 30 Dec. 2016].

39.
39
Olson, P. (2016). Duolingo's New Chat Bots Can Teach You French, German And Spanish. [online] Forbes.com. Available at:
http://www.forbes.com/sites/parmyolson/2016/10/07/duolingo-chat-bots-artificial-intelligence/#2d80c93766e5 [Accessed 2 Jan. 2017].
Ovchinnikova, E. (2012). Integration of world knowledge for natural language understanding. 1st ed. Amsterdam: Atlantis Press, p.15.
Pandey, R. (2016). Global ad spend: Trends for 2016 and 2017. [online] Marketing Interactive. Available at: http://www.marketing-interactive.com/global-ad-spend-trends-2016-
2017/ [Accessed 30 Dec. 2016].
Pearce, K. (2013). Ask Ivy. [online] Jobs@Intel Blog. Available at: http://blogs.intel.com/jobs/2013/05/16/ask-ivy/ [Accessed 2 Jan. 2017].
Perez-Marin, D. and Pascual-Nieto, I. (2011). Conversational Agents and Natural Language Interaction. 1st ed. Hershey, PA: Information Science Reference, pp.1-22, 107-127, 358-
378.
Publicis Groupe, (2016a). Publicis Groupe Announces Important Nominations and its Transformation Plan. [online] Available at:
http://livredor.publicis.com/notesInternes/2015/021215_EN_PublicisGroupe_Transformation_DEF.pdf [Accessed 20 Dec. 2016].
Publicis Groupe, (2016b). Third Quarter 2016 Revenue. [online] Paris: Publicis Groupe. Available at: http://www.publicisgroupe.com/media/display/id/8072.pdf [Accessed 31 Dec.
2016].
Publicis Groupe, (2016c). 2015 Registration Document - Annual Financial Report. [online] Paris: Publicis Groupe, pp.90 - 95. Available at:
http://www.publicisgroupe.com/documents/document-de-reference-2015-en.pdf [Accessed 31 Dec. 2016].
Richman, D. (2016). Microsoft Azure offers ‘bots as a service’ as demand rises for automated online interactions. [online] GeekWire. Available at:
http://www.geekwire.com/2016/microsoft-azure-offers-bots-service-demand-rises-automated-online-interactions/ [Accessed 5 Jan. 2017].
Riel, J. (2016). How Chatbots Can Help With Learning. [online] College of Education | University of Illinois Chicago. Available at: https://education.uic.edu/academics-
admissions/student-life/how-chatbots-can-help-learning [Accessed 2 Jan. 2017].
Ries, E. (2011). The Lean Startup. 1st ed. London: Penguin Group, pp.56-78.
Rosenberg, S. (2016). New Messenger Platform Features: Link Ads to Messenger, Enhanced Mobile Websites, Payments and More. [online] Facebook for Developers. Available at:
https://developers.facebook.com/blog/post/2016/09/12/new-messenger-features-payments-ads-enhanced-mobile-websites/ [Accessed 1 Jan. 2017].
Roxburgh, H. (2016). Why big consultancies buy design agencies. [online] Campaign Asia. Available at: http://www.campaignasia.com/article/why-big-consultancies-buy-design-
agencies/407973 [Accessed 30 Dec. 2016].
Sas.com. (n.d.). Machine Learning: What it is and why it matters. [online] Available at: http://www.sas.com/en_us/insights/analytics/machine-learning.html [Accessed 1 Jan. 2017].
Schaefer, M. (2015). 6 Reasons Marketing Is Moving In-House. [online] Harvard Business Review. Available at: https://hbr.org/2015/07/6-reasons-marketing-is-moving-in-house
[Accessed 30 Dec. 2016].
Schlicht, M. (2016). The Complete Beginner’s Guide To Chatbots. [online] Chatbots Magazine. Available at: https://chatbotsmagazine.com/the-complete-beginner-s-guide-to-
chatbots-8280b7b906ca#.dxwvm1po5 [Accessed 31 Dec. 2016].