Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Chatbots in HR: Improving the Employee Experience

8,458 views

Published on

Academic submission on the use of chatbots in HR to improve the employee experience. Includes the following topics: Origin and development of chatbots, why chatbots are of such interest now, applications and benefits of HR chatbots, chatbots for HR information services, considerations when building chatbots, etc.

Published in: Technology
  • 1 minute a day to keep your weight away! ●●● https://tinyurl.com/1minweight4u
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Do This Simple 2-Minute Ritual To Loss 1 Pound Of Belly Fat Every 72 Hours ▲▲▲ https://tinyurl.com/y6qaaou7
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • New gender study 74% of men are more attracted to women who do this one thing, read more ♣♣♣ https://tinyurl.com/y6enhezd
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • There are over 16,000 woodworking plans that comes with step-by-step instructions and detailed photos, Click here to take a look ◆◆◆ http://ishbv.com/tedsplans/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • There are over 16,000 woodworking plans that comes with step-by-step instructions and detailed photos, Click here to take a look ♥♥♥ http://tinyurl.com/yy9yh8fu
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Chatbots in HR: Improving the Employee Experience

  1. 1. Chatbots in HR Improving the Employee Experience Kong Yean Hwei, Amy | Advanced Work Based Project Report Crew 4 | Hyper Island Singapore | Jan 2017
  2. 2. 2 Table of Contents EXECUTIVE SUMMARY ……………………………………………………………. 2 WHAT ARE WE EXPLORING? ……………………………………………………. Changing Landscape in Advertising ……………….…………….…………… Digital is Part of the Process ……………………………………….……………. Why Chatbots, and Why HR? …………………………………………………… 4 4 4 5 EXPLORING THE LANDSCAPE – A LITERATURE REVIEW ……………… Origin and Development of Chatbots ………………………………………… Why are Chatbots of Such Interest Now? ……………………………………. Applications andBenefits ofHRChatbots …………………………………… Chatbots in PubComms SG for HR Information Services ………………. Designing and Building Chatbots ……………………………………………… Some Considerations When Implementing Chatbots …………………… Learnings and Conclusions for an Experiment …………………………….. 6 6 7 9 11 11 14 15 THE HR CHATBOT EXPERIMENT ………………………………………………. Experiment Considerations ……………………………………………………… Experiment Design ………………………………………………………………. Experiment Round 1 with Twine – New Insights …………………………... Experiment Round 2 with Chatfuel – Validation and More ……………. Overall Findings and Conclusion ……………………………………………….. Limitations of the Experiment ………………………………………………….. 16 16 16 18 24 28 30 REVIEW AND RECOMMENDATIONS …………………………………………… Yay or Nay? …………………………………………………………………………… How Much Does It Cost? …………………………………………………………. It Could Save SGD 125,000 Annually ………………………………………….. 30 30 30 30 Suggested Timeline and Next Steps ………………………………………….. Other Recommendations and Conclusions …………………………………. The Time is Now ……………………………………………………………………. 30 32 32 REFLECTIONS ………………………………………………………………………….. 32 REFERENCES ..……………………………………………………………………..….. 35 APPENDICES ..………………………………………………………………..……….. Appendix 1: Key Questions to Ask Before Investing in a Chatbot …… Appendix 2: Proposal to PubComms SG …….…………..……………….… Appendix 3: Qualitative Interviewing Framework …..…..…………....… Appendix 4: Twine Documentation on ERIS Prototype 1 .......……….. Appendix 5: Experiment Data and Analysis ………………………….…….. Appendix 6: Chatfuel Documentation on ERIS chatbot ………….…….. 41 42 ABOUT THE AUTHOR The author was most recently the HR Director in Publicis Communications Singapore. She held dual roles, leading HR operational responsibilities for Publicis Worldwide, MSLGROUP and Nurun, and the Learning & Development function for all brands within Publicis Communications Singapore, including the brands mentioned above plus Saatchi & Saatchi, Leo Burnett and Prodigious. This Advanced Work Based Project is based on Publicis Communications Singapore. REDACTED REDACTED
  3. 3. 3 EXECUTIVE SUMMARY Technology has disrupted the advertising and communications industry. Agencies are seeing a chronic decline in industry prices of 4.5% to 5% per year, compounded. Companies are trying to capitalize on this digital wave, including Publicis Groupe, the holding company of Publicis Communications,which has set this as a strategicpriority for the company. It is logical to assume that internal company processes should also reflect these digital priorities, and this Advanced Work Based Project aims to be part of the solution by exploring how technology applied to processes within Publicis Communications Singapore can provide value. Specifically, this report sets out to explore the following problem statement: “How might we explore the potential and applications of chatbots in HR, specifically in Publicis Communications Singapore, to improve the employee experience?” This report reviews the origin and development of chatbots, and the confluence of recent developments that have triggered the surge of interest in chatbots. It also reviews the potential applications of chatbots in HR and its benefits, and considerations needed for the design, development, and implementation of chatbots. An experiment using the Build-Measure-Learn feedback loop of the Lean Startup methodology was conducted in Publicis Communications Singapore to gain employee feedback on the problem statement. Two prototypes were tested with a small group of employees; one was a low- fidelity prototype using the Twine program, and the other a chatbot Minimum Viable Product (MVP) written using Chatfuel, a Do-It-Yourself chatbot builder. To access the ERIS MVP on Chatfuel, please email the author at amy.kong@hyperisland.sg for an access link. Based on the learnings of the literature review, the chatbot was developed as a text-based, rules- and retrieval-based, closed domain chatbot with some level of artificial intelligence. The topic of the chatbot was focused on providing information on a specific HR policy. Interviewees were asked to interact with the prototype, then semi- structured interviews were conducted to get their feedback on the prototype and their views on the usefulness and potential applications of chatbots in the HR function in PubComms SG. Overall results show that employees like chatbots’ ability to provide instant access to information, the ability for them to gain answers independently, and friendly tone. Each mentioned at least two other areas of applications of chatbots in HR, including providing information on company policies, processes, onboarding, and training. Additional insights that emerged included that interviewees felt chatbots would help them to avoid embarrassment when they needed to ask certain types of questions, and that chatbots could provide anonymity when they need to read up on sensitive company policies. Interviewees were clear that they would go to a chatbot for functional queries, while they would go to a HR team member for counsel on non- routine requests or emotional issues. This opens the possibility that chatbots could help to transition HR away from a transactional role towards an expertise role in the company. Based on the results of the experiment, there is significant potential and value in investing in HR chatbots to improve the employee experience and employee productivity. Calculations show that Publicis Communications Singapore could save about SGD 125,000 annually if just 5 minutes were shaved off each HR request from employees.
  4. 4. 4 WHAT ARE WE EXPLORING? Changing Landscape in Advertising Technology has disrupted most industries, and it is no different in advertising. Where advertisements previously lay predominantly in the domain and control of advertising agencies, the advent of technology has changed that. The industry has shifted dramatically and advertising agencies now face challenges on multiple fronts: • Emergence of new forms of advertising such as online, mobile and search engine marketing (SEM) that has seen a redistribution of budgets from traditional forms of advertising (Pandey, 2016) • Competition from companies that were never considered competitors before, such as o Management consultancies (Dan, 2016; Roxburgh, 2016; Vranica, 2016) o Publishers (Marshall and Alpert, 2016) o Digital tech companies (Johnson, 2015) o In-house agencies (Schaefer, 2015; Stiglin, 2015; Morrison, 2016) • Commoditization of the industry as technology enables anyone to become creative content creators (Fromowitz, 2016; Oetting, 2015) As a result, the industry is seeing huge pressure on prices and margins. According to Farmer (2015), declining fees (at 2-3% annually) and growing workloads (2-3%annually)overthepast 20years have resulted in a chronic decline in industry prices of 4.5% to 5% per year, compounded. Digital is Part of the Process Many companies, and industries, are trying to capitalize on this digital wave, including Publicis Groupe. Publicis Groupe is the holding company that owns Publicis Communications (PubComms), the Client brand that this Advanced Work Based Project (AWBP) is based on. In fact, it is of such a strategic priority to Publicis Groupe that it separately reports its digital revenues in its quarterly earnings reports, and current digital revenues stand at 54% of total Groupe revenue in the third quarter of 2016 (Publicis Groupe, 2016b). It is also investing in training and development efforts for its employees. In its 2015 Annual Report Registration Document (Publicis Groupe, 2016c), it states, “…the goal is to ensure that each employee is able to acquire basic know-how, whether in rudimentary coding or better understanding the latest generation applications. The boom in mobile device usage and the new challenges of interconnectivity (connected objects) have led agencies to hire and train talented individuals with multiple skills, given the speed of industry change… It is essential to support our future managers.” If employees are expected to be digitally-savvy and able to recommend leading edge digital communications solutions for clients, then it is logical to assume that internal company processes should reflect these digital priorities as part of the digital immersion for company employees. According to Edgar Schein (cited in Christensen et al, 2016),processes are a criticalpart ofan organization’s unspoken culture. They tell people inside the company, “This is what matters most to us.” However, there are some that say that despite all the rhetoric, things are not changing: “What’s broken is not our people. It’s our process. It’s a process that has been endlessly debated but not reinvented, and it has not adapted to the changing world around us.” Grayson and North, 2016
  5. 5. 5 “What’s broken is not our people. It’s our process. It’s a process that has been endlessly debated but not reinvented, and it has not adapted to the changing world around us... We traffic in risky ideas. But we don’t make any bold moves to change how we operate internally.” (Grayson and North, 2016) This Advanced Work Based Project (AWBP) aims to be part of the solution, by exploring how technology applied to processes in a company – Publicis Communications Singapore (PubComms SG) - can provide value. Specifically, the use of chatbots in HR. Why Chatbots, and Why HR? Chatbots are a hot topic now, but most of the current discussion seems to focus on the use of chatbots to engage with customers. One of the functions of Human Resources (HR), as a parallel, is like the employee customer service function of the company. So, would it be a far stretch to say that chatbots could also have significant potential in HR? Currently, most of the software or technology used in the PubComms HR function are used for non- employee facing activities (backend functions) to ensure operational continuity or as a management reporting tool, e.g. payroll systems, PTalent (Publicis Groupe’s HR Information System). There are currently few technologies used that facilitate or assist the interaction between the HR function and employees, and all interactions are manually managed, e.g. HR queries, notifications to employees on HR updates etc. Improving the employee experience cannot be underestimated. Advertising agencies are expertise- led companies, i.e. talent is a strategic differentiator, and so talent attraction and retention should be key activities for the company. This is even more important now, as new competitors like tech firms and startups are aggressively wooing talent away (Tadena, 2015; Ember, 2016). This represents an area of opportunity for exploration, to use technology to improve processes and help create a company culture that is aligned with its strategic priorities, that in turn improves the overall employee experience. As such, the defined problem statement that we will use in this paper is: “How might we explore the potential and applications of chatbots in HR, specifically in PubComms SG, to improve the employee experience?” ABOUT PUBLICIS COMMUNICATIONS BACKGROUND PubComms is one of 4 Solutions hubs within Publicis Groupe and comprises all of Publicis Groupe’s creative brand networks, including Publicis Worldwide, MSLGROUP, Nurun, Saatchi & Saatchi, Leo Burnett, BBH, Marcel and Prodigious (Publicis Groupe, 2016a). Publicis Groupe is one of the largest marketing and communications company in the world, alongside WPP, Interpublic, and Omnicom (Elliott, 2002). In early 2016, a Chief Executive Officer was appointed to lead Publicis Communications in Singapore (Mumbrella Asia, 2016), and subsequently a Chief Talent Officer and a Chief Finance Officer. The finance and HR functions across the various brands were consolidated to align with the new management structure, overseeing a headcount of about 250 people. As each creative brand previously had its own HR policies, the HR team (author included), was tasked to align and improve employee and HR operations and policies across the various brands under PubComms SG. CLIENT STAKEHOLDERS The key client contact for this project was Dan Spencer, Chief Talent Officer, Singapore and Australia/New Zealand, Publicis Communications. Other key stakeholders included the HR leaders of PubComms SG, Sharon Ooi, HR Director, and Adele Sam, Talent Manager, who provided the information and liaison support in the execution of the experiment. REDACTED REDACTED REDACTEDREDACTEDREDACTED
  6. 6. 6 EXPLORING THE LANDSCAPE – A LITERATURE REVIEW Origin and Development of Chatbots What is a Chatbot? Different terms have been used for chatbots (or chatterbots), including conversational agents, dialog systems and virtual assistants. This is due to the proliferation of a variety of similar systems built with different technical architecture (Perez-Marin and Pascual-Nieto, 2011), for different purposes. For this report, we define a chatbot as an automated online software program that tries to mimic a human conversation in its interaction with a human user to answer questions or perform tasks (Shawar and Atwell, 2007; Knowledge@Wharton, 2016). Specifically, we will focus this report on text-based chatbots. Chatbots can be built for websites or apps and accessed through smartphones, tablets, desktops and tablets. In recent years, most of the major messaging platforms have launched tools to help developers integrate or deploy chatbots on their platforms. These include Facebook Messenger, Slack, Telegram, Google and Microsoft, Kik, among others (Richman, 2016). What makes a Chatbot? For a chatbot to conduct human-like conversation, ArtificialIntelligence (AI) is a criticalcomponent.Nils J. Nilsson, as cited by Stone et al. (2016), defines AI as the “activity devoted to making machines intelligent, and intelligence is that quality that enables an entity to function appropriately and with foresight in its environment”. AI has many applications (Stone et al., 2016), but in this report, we will narrow the topics to those relevant to text-based chatbots. Chatbots functions in two ways (Schlicht, 2016): 1. Rule-based: These types of chatbots respond to very specific commands (rules), and are unable to process if information is not input in the right way. It is only as smart as you program it to be 2. Machine Learning: Machine Learning is a form of AI, and machine-learning chatbots continuously get smarter as they learn, via Natural Language Processing (NLP), from the conversations it has with people The recent boom in chatbots has been fuelled by advancements in AI research, combined with technological improvements such as the availability of cloud computing resources and wide-spread, web-based data gathering (Stone et al., 2016). These advancements in AI include the areas of: • Natural language Processing (NLP) and Natural Language Understanding (NLU) NLP explores how computers can understand and manipulate natural language text or speech to do useful things (Chowdhury, 2005). NLU is a subset of NLP that deals with machine reading comprehension (Ovchinnikova, 2012), i.e. the ability of the computer to understand human language. In short, NLP applications try to understand natural human communication and communicate in return (Marr, 2016). • Machine Learning With machine learning,computers areenabledto continuously learn from data on its own, using algorithms (set of rules or steps), to find insights without being explicitly programmed where to look (Sas.com, n.d.). In chatbots, machine learning is used to help machines learn and understand the vast nuances in human language, and learn to respond in a manner that the audience is likely to comprehend (Marr, 2016). While this report will not go into the details of these technologies, these are key terms to understand as they will be mentioned in any discussion around chatbots.
  7. 7. 7 Key Developments in AI and Chatbots Turing Test and Loebner Prize Competition One of the most significant milestones was created when Alan Turing created “The Imitation Game”, now known as the Turing Test, in 1950. Starting with the question “Can machines think?” (Turing, 1950), Turing created a test that would determine if a computer is capable of thinking like a human. The Turing Test has since been used as a benchmark that any AI must pass en route to true intelligence (Ball, 2015). However, it was not until 1991 that there was an implementation of the Turing Test (Loebner.net, n.d.b) – the annual Loebner Prize Competition – created by Dr. Hugh Loebner to advance the field of AI (Loebner, n.d.a). As long as there is an entry for that year, the prize will be awarded, and so while there have been multiple Loebner Prize winners, no program passed the Turing Test until 2014 (Griffin, 2014). The First Chatbot, ELIZA Then in 1966, the first chatbot ever coded, ELIZA, was documented by Joseph Weizenbaum. ELIZA was written to simulate a psychotherapist conducting a psychiatric interview (Weizenbaum, 1966). ELIZA worked by analysing the words users inputandmatching themto a list ofpossible scripted responses (Newman, 2016). More critically, ELIZA seemed so believable to users that Weizenbaum perceived his program as a threat, and in his book Computer Power and Human Reason, he attacked AI (specifically ELIZA) and computer science research, thereby slowing the pace of research into AI (Epstein, Roberts and Beber, 2008). A.L.I.C.E In 1995, A.L.I.C.E. (Artificial Linguistic Internet Computer Entity), the first AIML-based (Artificial Intelligence Markup Language) personality program, was created. AIML is a computer language designed for creating natural language-based chatbots that is still used today. A.L.I.C.E.won theLoebnerPrize three times,in 2000, 2001 and 2004 (Epstein, Roberts and Beber, 2008), and is one of the strongest AI programs of its time. It was programmed to give the illusion that it was intelligent and self-aware, but was driven by “supervised learning”, where a botmaster monitors the conversations and creates content to make responses more appropriate, accurate, “human”, etc (Epstein, Roberts and Beber, 2008). Bots for Messenger In April 2016, Facebook launched the bots for Messenger Platform (Marcus, 2016). While Facebook is not the first company to launch a bot platform, it is significant because it now provided, at scale, the opportunity for businesses to automate their one-to-one engagement with their customers to drive business and e-commerce (Rosenberg, 2016), thus driving the current interest and boom in chatbots. Figure 1 shows a more detailed timeline on the history of chatbots. Why Are Chatbots of Such Interest Now? A confluence of several developments has triggered the surge of interest in chatbots: Advancements in AI Research and Computing As mentioned, advancements in AI research, particularly in machine learning and NLP, are enabling chatbots to provide more accurate and intelligent responses. Improvements in computing and hardware technologies have also enabled integration of various apps into a chatbot that allow chatbots to complete tasks with little or no operator intervention. In fact, research firm Gartner predicts that in 2017, only 33% of all customer service interactions will need a human intermediary, as compared to 60% in 2014. (Gartner.com, 2015)
  8. 8. 8 Figure 1.: The History of Chatbots (Futurism, n.d.)
  9. 9. 9 Rise of Mobile Messaging and Fall of Apps According to Business Insider (2016), monthly active users of messaging apps (where chatbots can reside) surpassed social networks in 2015 in terms of monthly active users (see Figure 2). Companies also face increasing friction in getting consumers to download and use a mobile application. The average mobile user globally has 33 apps installed on his or her device, of which 12 apps are used daily. Of that, 80% of the average global mobile user’s time is spent only on 3 apps (Meeker, 2016). This means that companies and brands are finding it increasingly difficult to engage one-on-one with customers. App Integrations Attract Users to Stay Within Platform Eco-system The ability to integrate various apps into a chatbot also means that users are attracted to stay within the messaging app (Bayerque, 2016). In the context ofbrands,it means that customers are likely to stay within the brand eco-system, thereby increasing duration of interaction with the brand. Bots Can “PiggyBack” Existing Massive Platforms Bots are the beginning of micro apps on the backs of massive platforms which will lead to more focus and reach for start-ups and more delighted users (Batalion, 2016) because brands can: 1. Spend less time in development, e.g. building a chatbot on Facebook Messenger allows you to build one experience (without needing to write for different mobile phone sizes, or operating systems) and access Facebook user rich profiles instantly 2. Focus more time on creating positive user experiences because they can leverage richer user interfaces and software plug-ins (that platforms develop or offer) to process input What all this means is that companies and organisations can now increase quality one-to-one, personalised engagements with customers at scale, at potentially reduced costs and time. Applications and Benefits of HR chatbots Current Applications of Chatbots in HR General HR Support Information Services (Search and Retrieval) A typical challenge that employees face is knowing where and how to find information on HR policies and processes, or even whom to approach within the HR department for specific queries. Benefits of using chatbots in information services include (McNeale and Newyear, 2013): • Tackling the problem of users not familiar with specific terminology and jargon • Unlike humans, chatbots are unruffled by rude customers, high traffic, or repeated requests for the same information, and remain consistently patient and polite This also shifts the responsibility of locating the needed information from the user to the designer of the chatbot where the designer leads the user through a question and answer dialogue to discover the information needed and provide it (McNeale and Newyear, 2013). An example of such a chatbot is “Ask Ivy”, a “virtual HR agent” launched by Intel in 2013 that uses a combinationofNLP,AIandoptimizedsearchhosted on their Intranet to help answer employees' questions about their pay, stock, benefits, or other HR programs (Pearce, 2013). Figure 2.: Messaging Apps Surpass Social Networks (Business Insider, 2016)
  10. 10. 10 HR Task Automation or Assistance Virtual personal assistants are now a big focus due to the ability to now integrate several apps (and actions) into chatbots. The most current and well known examples are iPhone’s Siri, Amazon’s Alexa, Microsoft’s Cortana and the Google Assistant (Dunn, 2016). In the domain of virtual chatbot assistants in HR, there are some products now in development in the market, including: • HR task automation chatbot from ADP Innovation labs that is still in development (Boulton, 2016) • Mila, overstock.com’s chatbot created to assist with employee sick leave reporting and team rescheduling at their call center (Greenfield, 2016; Aspect.com, n.d.) Recruitment Streamlining Job Applications Chatbots can be used to vet potential candidates (Bhaduri, 2016). One recent example is Mya (http://trymya.io/), a chatbot that vets and interviews job candidates, using textual analytics to filter irrelevant resumes and interviews selected candidates on their work experience and qualifications. Mya also administers tests, provides application status updates and tips to candidates (Dishman, 2016; Deutscher, 2016; Trymya.io, n.d.). Q&A Tool for Job Applicants Chatbots can also serve as a Q&A resource for potential candidates on the position they are hiring for, or the company’s hiring policies (Sullivan, 2016). Benefits of using chatbots this way include: Higher quality of applicants (through candidates’ self- selection), improved candidate experience, reduce time spent by recruiters (and costs) in answering repeated FAQs from candidates. One of the most established examples is Sgt. Star, launched in 2007 by the United States Army to help potential recruits to learn about a career in the Army. First residing on the Army recruitment webpage,Sgt. Star then made its debut as a mobile app in 2014. (Maass, 2014; Nextit.com, n.d.). Onboarding Chatbots can also beusedto contactnew employees (or allow them to self-serve) with onboarding information, such as organisational structure and history, how to get enrolled in programs or benefits, suggestions on colleagues to meet, and links to the required HR forms, in place of a HR generalist (Boulton, 2016; Zwier, 2016). Training (Bhaduri, 2016) There is already ongoing discussion in academic and education sectors on the value of chatbots in training and learning that can be applied to the corporate sector (Perez-Marin and Pascual-Nieto, 2011; Olson, 2016). Chatbots can be used to deliver interactive and personalised training, monitoring and nudging, and instant help (Newton, 2016; Riel, 2016). Benefits of Using Chatbots in HR Based on the above, the potential benefits of using chatbots in HR include improving: • Employee productivity and employee experience by: o Providing on-demand employee engagement with 24/7, instant responses o Simplifying information access o Supplying just-in-time and fast access to information o Personalisation of the employee interaction • HR team’s performance (value-add, productivity and motivation) by o Handling repetitive tasks (e.g. answering FAQs) thereby - Freeing up HR resources to focus on more value-added work or reduce costs - Reduce frustration and/or monotony of the HR team o Automating standardised tasks o Identifying potential issues using data analytics and addressing them before they escalate
  11. 11. 11 Chatbots in PubComms SG for HR Information Services The experiment conducted to explore the problem statement of this report focused on the use of chatbots to provide HR information services in PubComms SG as it strikes the best balance between feasibility and desirability. Why Information Services? Firstly, the HR function in PubComms SG already has ready content that can be re-purposed to populate any chatbot prototype. Secondly, as a first investment in chatbots, the easiest chatbots to build are rules-based chatbots that rely on specific topics (closed domain) with predefined responses (retrieval-based) (Figure 3.). Having machine learning capabilities is a bonus and can be incorporated into the chatbot with subsequent iterations (Varagnat, 2016). Jumping into a chatbot that requires extensive machine learning capabilities from the start will create barriers of entry as it can be time-consuming (due to lack of expertise) and potentially costly. What About the Other Applications of Chatbots? While recruitment is a constant needfor PubComms SG, it is not feasible because: • In the communications agency environment, the company information that candidates would be interested in (e.g. client roster) is considered confidential. It is not available on public forums and usually only communicated face-to-face to the candidate verbally during the interview. The highly customised nature of remuneration in the advertising industry also does not suit the use of chatbots for recruitment Q&A at this point in time • Using chatbots to vet potential employees require sophisticated filtering technology as the interviewing process also involves interviewing for soft skills. The technology is still very new, and to demonstrate the potential of chatbots in PubComms SG, it is better to choose an area that is more established. Additionally, the various hiring managers within PubComms SG have different interviewing and selection criteria, and it would be difficult to standardise this quickly to build the prototype. Finally, most off-the-shelf products in the market are US- based and there may be cultural differences in the way resumes are written, or how candidates respond in an interview. In the area of HR task automation, companies openly admit that it is very early days and all these assistants are far from polished (Dunn, 2016). As for onboarding and training, there are currently no consolidated company information materials (due to the recent re-organisation) that can be used to populate the chatbot. Use cases of the value of chatbots in corporate training in the industry are also still few. Designing and Building Chatbots This section is split into the following subsections, to explore best approaches for designing and building chatbots: 1. What Consumers Want 2. Three Approaches to Creating Chatbots 3. User Experience Design Consideration REDACTED
  12. 12. 12 Appendix 1 also provides a list of key questions to consider prior to investing resources in a chatbot. What Consumers Want Prior to designing a chatbot,it is worth exploring the functions and experiences consumers want. As this is a relatively new area of interest commercially, there are no any large scale, benchmark global consumer surveys that have been conducted, but some surveys have been conducted in specific markets. These provide starting insights that could be validated through Proofof Concept experiments. Based on results of three recent consumer surveys, one in the US and two in the UK with sample sizes of 1,000 respondents each (Aspect, 2016; Mindshare, Goldsmiths University of London, 2016; myclever Agency, 2016), these are some of the findings grouped around certain central themes: “Humanness” of a Chatbot Consumers prioritize efficiency in understanding their queries over the personality or friendliness of the chatbot. This was raised in the two UK surveys. However, the US survey indicated that consumers expect chatbots to perform better in “friendliness” and “ease of use” as compared to “accuracy” and “interaction success”. These could be inter-related, i.e. consumers don’t have a good experience with the efficiency of chatbots, therefore prioritising them higher than the tone of interaction of chatbots. Expected Benefits • Getting 24-hour service • Quick answers to simple to moderate questions • Getting an instant response • Being able to self-serve without having to talk to a customer service agent Predicted / Expected Functions of Chatbots • Quick emergency answers • Forwarding to appropriate human when requested by user • Buy basic items (clothes, food) • Retain information on context and history of all previous interactions for a more personalised experience • Get basic information about a company, product, or service • Information on product availability Three Approaches to Creating Chatbots There are three ways to create chatbots (Vorobiov and Kotsurenko, n.d.): 1. Buy a ready solution: there are vendors who offer off-the-shelf solutions. Examples of HR- specific chatbot solutions include Mya, mentioned earlier, Talla for onboarding employees within Slack (https://talla.com/). This allows businesses to speed up time-to-market, but the downside is cost and not all ready-made solutions are customisable. 2. Use DIY chatbot builders: there are a variety of Do-It-Yourself (DIY) bot builders, some requiring technical knowledge, and others not requiring any. A higher degree of customisation is possible, and it is relatively cheap and quick. Functions may be limited but new features are constantly being added.However,it can be hardto compare functionalities and prices across chatbot builders as business models vary widely. 3. Build a chatbot from scratch: While technically still the most challenging, there are many developer tools available in the market that make chatbot building much easier now. Naturally, this option requires technical expertise and more cost, but results in a solution that is tailor-made for, and proprietary to, the company. Figure 4 shows an infographic ofthe current chatbot landscape. This infographic does not include chatbot development vendors as they are industry- specific, but it gives a good overview of the market. As the objective of this report is to explore the potential and applications of chatbots in HR to improve the employee experience, we will focus on using DIY chatbot builders (referred to as chatbot builders from here) as it is the cheapest and fastest way to build a prototype, and evaluating technical considerations is not an area of discussion here.
  13. 13. 13 User Experience Design Considerations Given that part of this report’s problem statement is concerned with improving the PubComms SG employee experience, we need to examine how to create good User Experience (UX) in a chatbot. Thisis especially whenchatbotsnowenableone-on- one interactions between the user and the company, the quality of that interaction becomes more important. Many articles use the terms UX and User Interface (UI) interchangeably. For clarity, we attempt to make a distinction between the two. The industry perspective of UX focuses on the “optimisation of a product for effective and enjoyable use”, including ensuring the product logically flows from step to step (Lo, 2014; Lamprecht, 2016). UI, on the other hand, is concerned with the “look and feel, the presentation and interactivity of the product”that guides theuser intoa goodinteraction with the product. To simplify it even further, the comparison between UX and UI: “Something that looks great but is difficult to use is exemplary of great UI and poor UX. While something very usable that looks terrible is exemplary of great UX and poor UI.” Helga Moreno, as cited by Lamprecht (2016) In this context, a large part of creating a good UI is already handled by chatbot builders as their products come with pre-set user interfaces. Some industry best practices to designing a good UX in chatbots (Ady, 2016; Howard, 2016; Jain, 2016; Yao, 2016): Smallest Number of Steps Plan the conversation flowto be efficient,so that the user reaches his goal in a few steps as possible. Always Provide a Path to the End Goal Give users cues to reach next steps that lead to the end point, so that they are not left hanging mid-way in a conversation without knowing what to do. Figure 4.: Chatbots Landscape (Medium, 2016)
  14. 14. 14 Decide Parameters of Information to be Stored If one of the aims of the chatbot is to provide a personalised experience, then information about the user needs to be retained / stored. Establishing parameters (or categories) of information to be stored helps to provide targeted or relevant information for users. For example, if there are different sets of employee benefits associated with different job titles in the company, establishing job titles as a parameter allows the chatbot to retrieve the different sets of employee benefits by the user’s job title. Set the Right Expectations • Explain the purpose of the chatbot • Explain what the chatbot can do • Explain how to interact with the chatbot • Set a first action for the user to take • Provide hints or prompters on the possible paths of exploration at the end of the onboarding Allow Flexibility of Exploration Allow users the flexibility to skip around options. Use Buttons to Supplement Free Text Input Particularly for closed domain chatbots, buttons provide more accuracy in interpreting users’ requests since the options are pre-defined. Clicking a button is also faster than typing text. Layer AI to Interpret User’s Requests Use AI to learn the various ways users phrase requests differently so that the chatbot can trigger the right response or action, e.g. “Hi”, “Hello”, “Wassup”, “Yo” should trigger a return greeting or opening message from the chatbot. Use Images or Graphics to Supplement Text Images or graphics are easier to understand than text explanations. Provide a Null Response If the chatbot is unable to understand what the user wants,a message shouldbe triggeredto indicate so. Non-responses from the chatbot where the user is left not knowing what to do next should be avoided. Tone Should Reflect the Brand Personality For example, the chatbot may want to use more casual, trendy words if it is targeted at millennials. Also consider the use of emoji or gifs or graphics appropriate to the chosen tone of voice. Help Users Get Help Provide options for users to get help outside of the chatbot as needed. This can include directing the user to documentation, or a human point of contact for further inquiries. Integrate with Existing Business Systems Enable chatbots to access CRM databases to allow a richer experience with users and avoid having the user repeat their history or user details every time they interact with the chatbot. Some Considerations When Implementing Chatbots Hype There is a lot hype currently around chatbots, such that the general public's expectations of what chatbots can do will exceed the reality of what they can actually do (Hobson, 2016). This means that in launching any chatbot, companies should take care to communicate clearly to users what they should expect from any interactions with that specific chatbot. The reality also is that not every bot needs to be sophisticated, and it will depend on the objective or outcome that the chatbot is built for. Ifit is meant as an informationalservice,there is very little point in building a bot to conduct a conversation with a user, when the bot is meant to be transaction-based (Newman, 2016). Companies that are considering implementing chatbots need to be clear about the objective or goal of the chatbot and evaluate the technologies or functions needed to achieve that, rather than include unnecessary user features that result in additional cost for the company.
  15. 15. 15 Benchmark for a Minimum Viable Product (MVP) Software companies are used to developing a MVP, or a minimum standard version of the product that consumers would be willing to buy, to test in the market. However, for a good user experience, the product will likely need to have more accurate NLP and information before a MVP can be developed, which may mean that chatbots could require more capital than a traditional web or mobile app, where good frameworks are more commonly available (May, 2016). As above, clarity in the goal of the chatbot is essential,to be able to strip down to a MVPfor users. Alternatively, companies can first pilot a chatbot in a very confined and specific area to minimise capital outlay, while not compromising on the UX of the chatbot. This is the approach that has been adopted in developing the chatbot prototype for PubComms SG, as seen in the next sections of this paper. Learnings and Conclusions for an Experiment Chatbots are not new, but advancements in AI and computing technology have enabled the development of chatbots that can understand users better, have more meaningful conversations and perform tasks more effectively and accurately. This has opened chatbots for commercial possibilities at scale,particularly chatbots on messaging platforms. There are multiple possibilities of applying chatbots in HR; specific to PubComms SG, it is recommended that the focus of any chatbot pilot be focused on providing information services on specific topics that allow a MVP to be developed quickly and cost- effectively for piloting with employees. Also, the best approach in creating one seems to be the use of chatbot builders to minimize cost, increase the speed of prototype development and remove the constraints of needing technical expertise. It also allows more effort to be focused on providing a quality user experience to help employees better visualise the value and possibilities of using chatbots in the HR function of PubComms SG.
  16. 16. 16 THE HR CHATBOT EXPERIMENT Experiment Considerations Demonstrate Proof of Concept (POC) PubComms SG has never implemented similar technology in its support functions, so it was important to gain enough feedback to demonstrate POC to the PubComms SG HR team for them to evaluate if resources should be invested in such a tool, and provide justification to higher management on its value. Lean Resources to Build the Experiment As there was no budget, the experiment would have to be built on free platforms. Platforms That Do Not Require Much (or Any) Coding Knowledge As the author has no background in coding, the chatbot would have to be built on tools that require little to no coding knowledge. As the objective of this experiment was to gain employees’ feedback, and not evaluate technical capabilities of chatbots, this would not pose a significant problem in the experiment. Topic where a Chatbot would Provide Noticeable Value To demonstrate the value of a chatbot to users, the topic selected would need to be one where it is not common knowledge, or of a complexity level that would require employees to seek information or help. Ease of Testing and Confidentiality As PubComms SG currently has no intranet, the experiment would have to be designed and hosted on external (and possibly public) platforms. As such, the topic selected for the experiment should have no confidentiality impact on PubComms SG. Speed The chatbot had to be created quickly as the experiment neededtobe builtandconductedwithin one month. Experiment Design Parts of the Lean Startup methodology (Ries, 2011) was used to design the experiment. Advantages of this methodology vs traditional research or product development methods include developing products that customers (employees) want, more quickly and cheaply, and making it less risky (Blank, 2013). If we were to view the proposed HR chatbot as a startup, there are three distinct stages (Figure 5.) and two periods of focus (Maurya, 2012): Stage 1: Problem/Solution Fit – Is there a problem worth solving? Stage2:Product/Market Fit – HaveI built something people want? Stage 3: Scale – How do I accelerate growth? The experiment was designed to explore and answer Stage 1 and 2; i.e. show validated learning that will help with presenting recommendations to the client. PROBLEM / SOLUTION FIT • Stage 1 PRODUCT/MARKET FIT •• Stage 2 SCALE ••• Stage 3 Focus: Validated Learning Experiments: Pivots Focus: Growth Experiments: Optimizations Figure 5.: Three Stages of a Startup (Maurya, 2012)
  17. 17. 17 Build-Measure-Learn Feedback Loop A key component of Lean Startup is the Build- Measure-Learn Feedback Loop (see Figure 6.) The aim of this is to go through the Feedback Loop in as little time and as few resources as possible to maximize learning through incremental iterative engineering and gain insights (Blank, 2015). It would allow the requiredminimum feedback to be gathered to demonstrate POC, enable the experiment to be conducted quickly, enable pivots on the idea, and help sharpen the final recommendations made to the client. It was also made clear to the client that this is was an exploratory pilot project, as evidenced by the Research Questions listed in the project proposal to the client (Appendix 2) where the expected outcome of the project was “A set of recommendations on the value and use of chatbots in HR to improve the employee experience…”. According to Ries (2011), planning for the Build- Measure-Learn (BML) Feedback Loop needs to be done in reverse, i.e. “we figure out what we need to learn … figure outwhatweneedto measure toknow if we are gaining validated learning, and then figure out what product we need to build to run that experiment and get that measurement”. As such, the subsequent sections explaining the experiment are elaborated in reverse of the BML Feedback Loop to demonstrate how this experiment was planned. Rounds of Experiments The original goal was to build a Minimum Viable Product (MVP) (Ries, 2011) for testing, but as this is a very new topic and the first time the author was exploring it, it was difficult to estimate the timeframe needed to build the MVP and the quality of the MVP. This made it challenging to manage client expectations on the output to expect. As a solution, it was then decided to set a minimum goal of producing a low-fidelity prototype, and a stretch goalofproducing an ERIS chatbot MVP.This was also in line with the Lean Startup methodology of gaining learnings and feedback as early in the process as possible. The client was provided with interim updates at the end of each experiment. Data Collection and Interviewee Selection Qualitative Interviews The prototype testing was conducted via qualitative interviews for the following reasons: 1. Target audience universe is small. Headcount for PubComms SG is approximately 250, and therefore any quantitative research would need relatively large numbers which was not feasible due to the time constraints 2. Main objective is learning. The purpose of the experiments was not just to validate or invalidate hypotheses, it was also to gain other insights that may help with any pivots (Ries, 2011, p. 178) which cannot be achieved with closed-ended, quantitative research Face-to-face, semi-structured interviews, i.e. interviews conducted following an interview guide using open-ended questions, were conducted among employees.Figure 6.: Build-Measure-Learn Feedback Loop (Ries 2011, p. 75) Ideas BUILD Product MEASURE Data LEARN Minimize total time through the loop REDACTED
  18. 18. 18 Employees were first given the prototype to try, and then asked for their reactions to the prototype and thoughts on the potential usefulness and applications of chatbots in the HR function of PubComms SG. Responses from interviewees were anonymised, coded and categorised for analysis. Using qualitative interviews as a research method would not only help to gain additional insights, but it would also ensure that there was no confirmation bias – defined as “seeking or interpreting evidence in ways that are partial to existing beliefs, expectations, or a hypothesis in hand” (Nickerson, 1998) – as the responses would be non-prompted. As with allqualitative data analysis,there is a certain levelofsubjectivity as interviewee responses are not standardized, and there is a level of judgement and interpretation required to code the data. Interviewee Selection A small sample of employees, 6 employees for each round of experiment (12 individuals across two rounds of experiments) was chosen. To ensure a fair representation of employees and reduce sampling bias (Uwex.edu, n.d.), the client selected the employee interviewees for each round based on a provided profile matrix (Table 1). The client was also asked to choose employees from a mix of job roles and brands. Eventually, 11 of the 12 selected interviewees responded within the response timeframe and were interviewed. Experiment Round 1 with Twine – New Insights Learning Objective: Value Over Growth According to Ries (2011, p. 81), the riskiest elements of any plan, are “leap-of-faith” assumptions, because the success of the venture rests on them. Two of the most important are the value hypothesis and the growth hypothesis. The value hypothesis tests whether a product or service really delivers value to customers once they are using it, and growth hypothesis tests how new customers will discover a product or service (Ries 2011, p. 61). For this experiment, the value hypothesis was chosen over the growth hypothesis for testing as this is the problem statement we were trying to answer, andemployees wouldhaveno problems discovering the chatbot as there are internal communication tools to publicize the availability of this tool. Going back to the problem statement, “How might we explore the potential and applications of chatbots in HR, specifically in PubComms SG, to improve the employee experience?”, the experiment sought to find interviewees’ feedback on two aspects: 1. Potential of HR chatbots in PubComms SG to improve the employee experience 2. Applications of HR chatbots in PubComms SG to improve the employee experience If these two statements yielded positive results, then it would support a client recommendation to invest in one. Measure: Using Literature Review as a Starting Point Using the learnings from the Literature Review as a starting point, the metrics of the experiment were defined as non-prompted responses from interviewees on the following: 1. Potential of HR chatbots in PubComms SG to improve the employee experience o 24-hour service o Quick answers o Getting an instant response o Ability to self-serve without having to talk to a customer service agent o Get basic information on the policy
  19. 19. 19 2. Applications of HR chatbots in PubComms SG to improve the employee experience o Ability of interviewees to envision 2 or more uses of chatbots in the HR function Two assumptions were made: • Non-prompted feedback from interviewees would reflect their priorities and desired benefits from chatbots • If interviewees perceived that HR chatbots provided value, they would be able to envision multiple possibilities for its application Other Measurements and Data Other questions were also developed to mine information that would help in exploring other areas of opportunities (see Appendix 3 for the Qualitative Interviewing Framework that was used as a guide to conduct the interviews). This included seeking information from employees on: • Product features anddesign thatemployees like, that would help with subsequent iterations of the chatbot • Other employee needs that could be solved, that would help the HR team to improve the overall employee experience • Other technology tools that employees may desire for use in PubComms SG that may help identify other areas of opportunity for technology investment in the company • Their perceptions of the role of chatbots within the HR team, to better understand how best to deploy technology and human resources, from an end-user perspective There were also other questions embedded into the interview for the benefit of the client, particularly two questions: 1. How would you describe the quality of your interactions with the HR team? 2. How would you rate the HR team’s use of technology to provide HR services to employees? These questions had no significant impact on the experiment, but were included to help the client better understand employees’ context and perspective of employees’ views of the HR team. Build: Focusing on the Conversational Flow Topic The selected topic for the chatbot was the Employee Referral Incentive Scheme (ERIS), where employees stand to receive a monetary incentive if they successfully refer their friends for open job positions in Singapore. ERIS was chosen because it is: o A defined area of inquiry that would provide a clear scope for the chatbot o A relatively new policy that employees still require assistance in understanding o Not too simple that employees find it easier to refer to policy documents o Complex enough for the chatbot to provide noticeable value-add to employees, but not exceeding the author’s technical capabilities and resources in creating the chatbot o Of a low level of confidentiality that would not impact PubComms SG should the prototype need to be hosted on public platforms, e.g. Facebook Messenger Software The low fidelity prototype was a html version simulating a chatbot conversation, built using the Twine program (https://twinery.org/). Originally created to write online choose-your-own-adventure games, it is a useful tool in designing and planning the conversational flows in chatbots because it allows for interactive, non-linear flows, like a conversation, without requiring the creator to have technical coding or programming expertise (Init.ai Decoded, 2016; WillowTree, Inc., 2016; Winstead, 2016; Vaneseltine, 2014). Twine was chosen as it provides an overview of the conversation flow (Figure 7), to allow the author to evaluate the effectiveness of the conversation, e.g. help users to get to the end point as fast as possible. Refer to Appendix 4 to access and view the ERIS conversational flow written on Twine (requires download of the free program to view). Video demonstrations of the front-end and back-end view are also included in the same appendix and shown in Figure 8 and 9.
  20. 20. 20 Figure 7.: ERIS – Twine Conversational Flow REDACTED
  21. 21. 21 Insights surfaced in the literature review section on good chatbot UX were incorporated. This includes ensuring that the prototype: • Set the right expectations • Allow flexibility of exploration • Create the tone to reflect the brand personality • Help users get help However, as Twine does not have certain functionalities of chatbots, the following best practices were not incorporated: • Use buttons to supplement free text input • Layer AI to interpret user’s requests • Use images or graphics to supplement text • Introduce product features gradually • Provide a null response • Integration with existing business systems • Securing information security and privacy of information This prototype was shown to interviewees on a desktop browser, rather than through a messaging platform. Results: New Insights Potential of HR Chatbots in PubComms SG Interview results show that the hypothesised benefits of chatbots to improve the employee experience were validated. Interviewees said that they thought that the chatbot provided good information on the policy and was personable. They also said that a chatbot would provide instant answers and enable employees to find information independently. Employees also rated the usefulness of a chatbot in helping to improve the employee experience 7.6 out of a possible 10, where 10 is the most useful, and 1 the least useful. Some interesting insights emerged from the interviews: Chatbot is easy to use ERIS is a relatively more complex HR policy within PubComms SG due to differing incentive amounts (depends on the type of job position employees are making referrals for), and incentive payment processes (depends on the nature of the employment contract of the referring employee). The conversational flow in the prototype was designed based on conditional logic, i.e. users were taken through a series of questions, and based on their answers, the chatbot would provide information that was specificto their situation (refer to Appendix 4 for documentation on the Twine program). Therefore, this feedback from interviewees, especially when it was the most mentioned feedback, is surprising. This indicates that a chatbot has the potential for simplifying complex information for users, if planned well. Chatbots help employees to avoid embarrassment An unexpected feedback from employees was that chatbots would help employees to avoid embarrassment when they needed to as HR questions that they felt were trivial. Chatbots help to provide anonymity This was raised specific to situations when colleagues are trying to understand policies on sensitive issues: “In an Asian culture, people are more non-confrontational, and may not want to raise alarm bells until they are sure. Making such information readily available without having to ask a HR team member [for access] allows them to do their initial research”. “In an Asian culture, people are more non-confrontational, and may not want to raise alarm bells until they are sure. Making such information readily available without having to ask a HR team member [f0r access] allows them to do their initial research.” REDACTED
  22. 22. 22 Figure 8: ERIS – Demonstration of Twine Front End User View VIDEO REDACTED
  23. 23. 23 Figure 9: ERIS – Demonstration of Twine Back End View VIDEO REDACTED
  24. 24. 24 Chatbots help to relieve the HR team Beyond seeing the benefits for themselves, most saw the benefit of using chatbots to help relieve the HR team’s workload so that they can redirect their time to focus on more important issues. Applications of HR Chatbots in PubComms SG All interviewees mentioned at least two other areas of applications, including: • Company policies • Company processes • Company information • Employee contract details • New employee induction / orientation • Training • IT requests Interestingly, the most of them related to information services. Refer to Appendix 5 for the full results and analysis of the interviews. UX Feedback Some areas of improvement that interviewees suggested to improve the user experience: • Reduce the wordiness of the chatbot • Have a “home” button (there was a “back” option in the prototype, but not a “home” option) Role of Chatbots vs HR team Interviewees were also clear on the role of the chatbot within the HR team. They would use a chatbot for functional or transactional queries and/or generic FAQs, and approach a HR member when they need counsel on non-routine requests, emotional issues, or exceptions to the regular policies and processes. This seems to indicate that chatbots can help transition HR away from a transaction perception in employees towards an expertise role. Experiment Round 2 with Chatfuel – Validation and More Learn: Would Feedback be the Same? Based on the learnings from Experiment Round 1, the following decisions were made on the desired learning from Round 2: • Potential of HR Chatbots in PubComms SG - To reconfirm the benefits even after the prototype is changed from a html, desktop version to a chatbot MVP hosted on a mobile messaging platform - Continue to listen and monitor if the new insights would continue to be mentioned in Round 2, thereby validating them • Applications of HR Chatbots in PubComms SG - Reconfirm the desired applications of chatbots in HR Build: Creating a chatbot MVP The aim of the Round 2 Experiment was to be able to test an ERIS chatbot MVP. The MVP was written using Chatfuel (http://chatfuel.com/), a chatbot builder. Two other bot builders were considered – Botsify and Wit.ai – but Chatfuel was eventually chosen as it had the best combination of elements that met the design considerations and required features of the experiment (Table 2).
  25. 25. 25 Figure 10: ERIS Chatbot Demonstration on Facebook Messenger VIDEO REDACTED
  26. 26. 26 Figure 11: ERIS Chatbot – Chatfuel Demonstration VIDEO REDACTED
  27. 27. 27 A key factor in deciding which chatbot builder to use was the messaging platform through which users could access and interact with the chatbot. Offering a chatbot through an owned PubComms SG asset was not possible as PubComms does not have any SG-specific intranet or website. A quick review of the chatbot builders available in the market show that most of them design their services to be hosted on a few key messaging platforms (Levinson, 2016; BotStory, 2016): Facebook Messenger, Slack, Skype, Telegram and Kik. As Facebook Messenger was the platformlikely to be most used among PubComms SG employees on their mobile phones, it was one of the criteria used to narrow the list of chatbot builders. Given the above, an additional question was added to the interview guide on interviewees’ thoughts on accessing a HR chatbot via Facebook Messenger. In adapting the Twine prototype to Chatfuel, these additional features were added to the MVP: • Include a ‘home’ button or its equivalent • Provide a null response • Allow users to type in their own queries / open- ended questions • More efficient user / conversational flows, i.e. less steps to get to the required information • Reduce wordiness and use more graphics • Layer AI to interpret users’ requests • Introduce product features gradually (Dropbox) Features recommended in the literature review that were not included: • Integration with existing business systems • Securing information security and privacy of information This was not required as the MVP was designed to be a standalone prototype (so it could be built quickly) and thus not needing additional information security measures. Appendix 6 includes source files for the two video demonstrations of the ERIS chatbot MVP you see in Figure 10 and 11. One shows the user view when interacting with the ERIS chatbot via Facebook Messenger, and the other is a demonstration of the architecture of the ERIS chatbot on Chatfuel. *To access the ERIS chatbot MVP on Chatfuel (backend view), please email the author at amy.kong@hyperisland.sg for an access link (valid for 24 hours). To interact with the ERIS chatbot, please use the Facebook Messenger app and search for @PubCommsERIS (case-sensitive) or scan the Messenger Code (Figure 12.) Results: Validation and More Interview results continued to validate the hypothesised benefits of chatbots to improve the employee experience were validated. In fact, the usefulness rating of chatbots in improving the employee experience rose from an average of 7.6 in Round 1 to 7.9 in Round 2. As this group of interviewees did not interact with the Twine prototype, there was no opportunity to investigate if this increase in rating was due to the switch in interaction from a desktop web browser to Facebook Messenger. Figure 12: ERIS Chatbot Messenger Code To interact with ERIS: 1. Launch the Facebook Messenger App 2. From Home, tap the icon 3. Tap your picture at the top of the page 4. Tap Scan Code 5. Scan the image on the left REDACTED REDACTED
  28. 28. 28 Interviewees (except for one) were also able to envision at least two other areas of applications of chatbots in HR, and were still able to differentiate when they would use a chatbot and when to approach a HR team member. A more nuanced view emerged on the feedback on avoiding embarrassment. Interviewees said that chatbots wouldhelp when employeesneededto ask: o Trivial questions (same feedback in Round 1) o Questions the employee is supposedto know the answers to o Questions the employee perceives may impact their image/reputation among co-workers Three additional areas of applications of chatbots in HR raised by interviewees in Round 2 include: • Making company announcements • Collecting employee suggestions • FAQ facility for hiring managers, e.g. if the company is still within the ‘S pass’ (a type of employment pass in Singapore) government quota for them to be able to hire foreigners The wordiness of the chatbot continued to be a consistent feedback for improvement in Round 2. Balancing between wordiness and omission of critical information will need to be considered, particularly in HR, where there could be legal or employment implications if important information is not communicated to the employee. Interviewees agreed that a HR chatbot needs to be hosted outside of Facebook, as it is a personal social media platform and HR chatbots are for professional purposes. One employee pointed out that the chatbot should be hosted on a platform that most employees use, and another pointed out that most employees use mobile phones for work, which makes a messenger- based chatbot appropriate. Overall, interviewees in Round 2 seemed to show more enthusiasm over the prototype. There were more expressions of excitement over the ERIS chatbot MVP. Some quotes from the interviewees: “Good that we are using technology in HR. We do it in our business. The fact that we are taking HR online makes it exciting. Now that I know that PubComms SG is considering it, I really want it and am excited for it. I can’t see how it would be a disadvantage.” “This is good. It primes employees for more future forward communications. That’s how it will be in the next few years. It will help people to learn how to apply it in their own (client) brands.” “We need to do things faster now, and something like this helps to get things done faster, especially when teams are so lean now.” “It is overwhelming how smart it is…I like the language used. It is friendly and approachable, and it makes me feel like I can ask any question.” “We don’t have to wait for a response from HR or be dependent on the HR person being around. Currentlywe pass a lot ofmessages around… so that HR doesn’t receive repeated questions. This allows you to find the answer directly yourself.” Overall Findings and Conclusion Problem Statement Exploration and Validation After two rounds of experiments, results indicate that employees think that chatbots have the “This is good. It primes employees for more future forward communications. That’s how it will be in the next few years. It will help people to learn how to apply it in their own (client) brands.”
  29. 29. 29 potential to improve the employee experience, and can be applied in various areas in HR within PubComms SG. Specifically, the benefits to employees include: • Instant access and answers • Ability to find information independently • Provide good information • Personable Interviewees found chatbots useful, and were able to envision how chatbots could be used in the HR function, including in: • Information services: company information, policies, processes, and status updates • Training • Information collection and dissemination: broadcasting company announcements and collecting employee suggestions and feedback Interviewees even suggested that chatbots could be used to log IT service requests. Finally, PubComms SG HR chatbots should be hosted on appropriate messaging platforms suitable for work use. Additional Insights • Chatbots seem to be able to play a role in simplifying complex information for employees, when the conversational flow of the chatbot is planned well • Chatbots help employees to avoid feeling embarrassed when asking certain types of questions • Chatbots can provide employees with a sense of anonymity on sensitive issues • In the longer term, chatbots could provide the opportunity to shift perceptions of the HR team from an administrative / transactional role to more of an expert role Limitations of the Experiment Small Sample As the experiment was designed to generate qualitative feedback, it also meant that the sample size is small and may not be representative of PubComms SG, although some of the feedback was validated in Round 2 of the experiment. Lack of Quantitative Metrics As data collected was qualitative, there were no quantitative metrics that were collected. One quantitative metric that could provide valuable insight (and justification to management) if future experiments are conducted, is the time saved in using a chatbot to complete HR tasks, versus the current approach. Confined Area of Enquiry The chatbot was built based on ERIS, which is a very specific HR policy within PubComms SG. In implementing any HR chatbot, the areas of enquiry would need to be broader than one specific policy, and the chatbot was not tested on its ability to filter or direct questions down different topic paths. This may impact the “ease of use” feedback that was seen in this experiment. “We need to do things faster now, and something like this helps to get things done faster, especially when teams are so lean now.” “Good that we are using technology in HR. We do it in our business. The fact that we are taking HR online makes it exciting. Now that I know that PubComms SG is considering it, I really want it and am excited for it. I can’t see how it would be a disadvantage.”
  30. 30. 30 Review and Recommendations Yay or Nay? Based on the Literature Review and Experiment, there is significant potentialandvalue in investing in HR chatbots to improve the employee experience and improve employee productivity by reducing time spent on completing HR tasks. The availability of chatbot builders and vendor solutions in the market now also lower the barriers of entry for any investment in building chatbots. How Much Does It Cost? It depends on the complexity of the chatbot. There are also still no industry standards for pricing, and various vendors and platforms have different business models which make it hard to do a cross- vendor or solution comparison of costs. However, to help with understanding the cost implications, Table 3 provides a list of budget items to be considered. It Could Save SGD 125,000 Annually Based on conservative estimates (see Table 4), a chatbot could save the company SGD 125,000 annually, excluding the cost of building the chatbot. This is a conservative estimate as a chatbot’s instant response capability also allows employees to reduce multi-tasking as a result of waiting for responses from the HR team. According to the American Psychological Association (2006), “even brief mental blocks created by shifting between tasks can cost as much as 40 percent of someone's productive time.” If you add in task automation capabilities into the chatbot, cost saved would also increase. Suggested Timeline and Next Steps It may be more appropriate to look at a timeline of 6 months to launch an “official” HR chatbot for the following reasons: Allow Time for Market to Mature The market is still at a relatively early stage of development. This should settle down somewhat in 6 months for PubComms SG to get a clearer perspective of the real possibilities of chatbots. Allow Time for More Vendor Solutions Emerge and Stabilize PubComms SG should also consider off-the shelf solutions, but many of these solutions are US-based, or still in beta versions. Waiting 6 months will allow more vendor solutions to emerge and become more stable. Also, given the huge messaging usage in Asia, Asia- specific or global solutions should emerge in the market soon. Other Decisions Around the Implementation of the Chatbot There are some areas that need to be reviewed and decided, which will influence how the HR chatbot in PubComms SG will be implemented. This includes:
  31. 31. 31 Topic/s of the Chatbot While the prototypes were built based on ERIS for specific reasons, ERIS is still a policy that is relevant only to a subset of employees, used infrequently. It may be worth considering a different set of policies to build the chatbot around, for example, top 10 FAQs, leave entitlements, medical benefits, list of panel of doctors etc. Choice of Messaging Platform Due to the confidentiality of HR policies and information, an appropriate messaging platform needs to be selected. At the same time, the choice of platform needs to be easily accessible by employees via their personal mobile devices. Some options that can be considered: • Skype for Business (Lync), the official corporate messaging tool for PubComms SG. This assumes that this feature will be made available to all employees on their mobile phones • Create a mobile-friendly intranet to house company information common to all brands, and also use it to host a chatbot Integration with Publicis Groupe Enterprise HR Software, Data Protection and Security A decision needs to be made on how personalised the chatbot should be. Personalising interactions requires integrating the chatbot with PubComm SG’s existing enterprise HR software to access employee data, and as such issues of data protection and security need to be addressed. On the upside, SAP, the enterprise HR software vendor used by Publicis Groupe, is partnering Kore to integrate chatbot products into its software (Besser, 2016), which resolves the security issues. On the downside, the speed at which chatbots can be deployed in PubComms SG may be slowed, as any use of the enterprise HR software will need to be authorised by the Pubicis Groupe global team, and vetted and handled by ReSources IT (Publicis Groupe IT department). Analytics There is an opportunity for PubComms SG to collect data to monitor and improve the performance of the chatbot, understand employees concerns better and eventually shape the Talent strategy of the company. The type of data to be collected should be considered during the design of the chatbot. This was not a focus for the experiment as the sample size was too small to collect any meaningful data. Employee Privacy and Anonymity Given that some employees have said that they feel that chatbots can provide anonymity, there needs to be discussion on how much,andwhat data should be collected, and what level of anonymity will be provided. Accuracy and Completeness of Information Scenarios where information provided could have a financial impact to the employee or company, or if an employee’s critical employment decision is dependent on a complete understanding of a specific HR policy, need to be discussed to ensure that proper check mechanisms are put in place. No doubt, these situations happens now without the use of chatbots, but they need to be discussed and the handling approach agreed on to ensure that there is a proper resolution mechanism in place. REDACTED REDACTED REDACTED
  32. 32. 32 HR Talent HR employees with a new skill set may be needed to drive a technology project such as this. The selected employee may not need to be a technical specialist, but needs to be able to understand the project parameters and work with vendors. In the longer term, a review will be needed of the composition andskillsetoftheHRteamasmoretechnologytools are used in the HR function. Other Recommendations and Considerations While there are many Do-It-Yourself chatbot builders in the market, it may be best for PubComms SG to hire a vendor to develop one if a customised solution is desired. Integrating the chatbot with the enterprise HR software will need considerable technical skill. Additionally, new chatbot features are constantly being added, and unless an employee is dedicatedto focus on building the chatbot, it would be difficult to keep up with the constant upgraded capabilities of the chatbot builders. The Time is Now Even as the suggested timeline is in 6 months’ time, the preparatory work needs to start now, as some of the considerations listed will involve careful consideration and multiple stakeholders that need time to resolve. MVPs can still be tested with a curated employee group of early adopters so that learnings are accumulated and can be baked into the “official” chatbot when ready. Acknowledgements My journey at Hyper Island would not have been possible without the advice and constant moral support from friends and family. I would like to specially thank the following who have contributed directly to my AWBP journey: Dan Spencer Chief Talent Officer, Publicis Communications, Singapore and Australia / New Zealand Leong Yeng Wai Student, Hyper Island Singapore Benjamin Koe Product Lead, Dentsu Aegis Global Data Innovation Centre Narendra Nag Chief Digital Officer, Xynteo Lim Chwen Yiing and Lim Han Boon Best Buddies Jonathan Briggs Co-Founder and Singapore Academic Director, Hyper Island Jarrod Howe Operations Director, Hyper Island Singapore
  33. 33. 33 Reflections This journey leading up to the completion of this Advanced Work Based Project (AWBP) has been quite an interesting and enlightening one. These are some of the learnings and experiences that have shaped this AWBP: Complexity of Chatbots and The Landscape Much of the literature talks about how easy it is to create chatbots. It is true, but it is also false. It is true that many of the tools now in the market, particularly the chatbot builders, are relatively easy to use. However, before getting to the pointofbuilding a chatbot,company decision makers need to first understand if they are making the right choice in investments, which means understanding the landscape, and that is where it gets complex. When I first started this project, I had no idea where to start, as I have no coding or IT specialist experience and my experience with technology is purely from an end-user perspective. As I started the research, it seemed like falling down a never-ending rabbit hole. Just to understand chatbot literature that is regularly peppered with jargon, you must know how it fits vis-à-vis AI, machine learning, NLP, NLU, Natural Language Generation. What’s the difference between a chatbot and a conversational agent? What is the difference between a conversational agent and a dialog system? That’s just understanding chatbots andits possible functions.To create the chatbot, you need to understand the different approaches to building one, then understand the comparative differences across the plethora of tools to choose one to use. And the list goes on. I first worked on wit.ai to start building the chatbot as recommended by chatbot-related articles as being easy to use. But it turned out to be an easy tool for developers to use, and not for non-coders like me. So, I had to abandon the work I started in wit.ai and re-start the search for a chatbot builder. For a HR teamthat may not have a large IT departmentto rely on for advice, or even a for CEO who is interested in investing in one, the landscape can be hard to decipher. Jargon Everywhere The proliferate use of jargon is only part of the problem. The other part of the problem is that at this stage of market development, terminology is not clearly defined or differentiated, or writers have a poor understanding of chatbot technology. Sometimes the same terms mean different things, or different terms mean the same thing. Some articles refer to AI as different from machine learning. Some articles talk about chatbot services while others talk about chatbot platforms when they mean the same thing. As such, I tried to be as clear as possible in this report on the different terms and definitions, and explain the chatbot landscape as simply as I could, based on my understanding. As this report is meant to be shared with PubComms SG, hopefully this will help provide clarity for the client. Easy for Whom? It is easy to use a chatbot builder, but it may not be simple to build a chatbot, or at least build a good one. As I was using wit.ai, I was not able to understand how the conversation with the user would flow (that would help me evaluate its effectiveness), as the tool does not help you to do so. This is the same with other chatbot builders that I explored. Adding on to that stress was the lack of time to conduct the experiment,which was how I then researched and found Twine. Even as I was using Twine, I realised that you need an understanding of conditional logic. Upon consulting my ex- colleague who has developer experience, I also realised that you need to plan the conversational flow not in sequential steps, but in blocks of information, and then create paths or flows to the informational blocks. REDACTED
  34. 34. 34 Reflections (cont’d) In short, anybody can build a chatbot, if that anybody understands how to plan conversational flows, understands conditional logic etc. Along the way, you’ll encounter other terms like ‘rails’, ‘conversational trees’, ‘dialog trees’ and so on, and down the rabbit hole you go again. Nevertheless, it is true that the barrier to entry for building chatbots are now far lower than before. Drinking From a Firehose Chatbots are such a hot topic now, that news articles and blog posts are generated almost every day. I found it difficult to keep track of the emerging articles and evaluate them for inclusion in this paper. It was also a challenge to select a balance of commercial vs academic sources. Firstly, academic journals do not necessarily reflect the current state of business discussion around the topic, and secondly, they tend to be very technical. There was significant temptation to use data primarily from commercial articles as they provide quick and short summaries. I have tried to refer to academic sources in this paper for key definitions, terms, concepts and key milestones and discussions, and kept the references to commercial articles confined to areas where the academic journals may not touch upon. Iteration vs Documentation When using Chatfuel, changes to any part of the chatbot is reflected ‘live’. There is also no option for version downloads or control. As such, during the experimentation phase, I found it difficult to keep track of the changes that I made to the chatbot. In the longer term, it also raises questions on how to restore chatbots should there be any changes that need to be reversed. When is Customer Feedback Valid? Part of the iteration process is seeking qualitative feedback from customers. One of the questions that I kept asking myself was, “at what point is customer feedback considered valid or an outlier?”. Ultimately, I made the decision based on what I felt made sense and was in line with the overall objective of the chatbot, but it seemed very subjective to me. If given the opportunity, I would ask current Lean Startup practitioners how they make such decisions, particularly with qualitative testing with small sample sizes Challenge in Integrating with Enterprise HR Software I anticipate that there will be challenges in getting this chatbot project running in PubComms SG if backend integration with the enterprise HR software is needed. Such requests will need to be approved by the Global Publicis Groupe team, and actioned by ReSources IT, which is the internal IT shared services function. This affects the speed at which backend functions in PubComms SG can capitalise on emerging opportunities. PubComms SG may be better off creating a generic chatbot now, then present the case study to the Global team for endorsement, to increase the speed of implementation. REDACTED
  35. 35. 35 References Ady, M. (2016). How to Build a Chatbot Your Users Will Love. [online] VentureBeat. Available at: http://venturebeat.com/2016/08/17/how-to-build-a-chatbot-your-users-will-love/ [Accessed 3 Jan. 2017]. American Psychological Association. (2006). Multitasking: Switching costs. [online] Available at: http://www.apa.org/research/action/multitask.aspx [Accessed 5 Jan. 2017]. Aspect, (2016). 2016 Aspect Consumer Experience Index. [online] Phoenix. Available at: https://www.aspect.com/globalassets/2016-aspect-consumer-experience-index- survey_index-results-final.pdf [Accessed 1 Jan. 2017]. Aspect.com. (n.d.). Aspect Mila: The Agent’s Personal Assistant. [online] Available at: https://www.aspect.com/solutions/workforce-optimization/aspect-mila [Accessed 2 Jan. 2017]. Ball, P. (2015). The truth about the Turing Test. [online] Bbc.com. Available at: http://www.bbc.com/future/story/20150724-the-problem-with-the-turing-test [Accessed 31 Dec. 2016]. Batalion, A. (2016). “Bot” is the Wrong Name.. and Why People Who Think They are Silly are Wrong. [online] Medium. Available at: https://medium.com/lightspeed-venture- partners/bot-is-the-wrong-name-and-why-people-who-think-they-are-silly-are-wrong-dc0c0b76ae18#.y4x0znitc [Accessed 1 Jan. 2017]. Bayerque, N. (2016). A Short History of Chatbots and Artificial Intelligence. [online] VentureBeat. Available at: http://venturebeat.com/2016/08/15/a-short-history-of-chatbots-and- artificial-intelligence/ [Accessed 1 Jan. 2017]. Besser, L. (2016). Kore bots for SAP empower every employee to delegate like a boss. [online] Ideas.sap.com. Available at: https://ideas.sap.com/D32372 [Accessed 4 Jan. 2017]. Bhaduri, A. (2016). The Digital Tsunami: HR. [online] Abhijit Bhaduri's Official Website. Available at: http://www.abhijitbhaduri.com/index.php/2016/08/the-digital-tsunami-hr/ [Accessed 2 Jan. 2017]. Blank, S. (2013). Why the Lean Start-Up Changes Everything. [online] Harvard Business Review. Available at: https://hbr.org/2013/05/why-the-lean-start-up-changes-everything [Accessed 22 Dec. 2016]. Blank, S. (2015). Why Build, Measure, Learn – isn’t just throwing things against the wall to see if they work – the Minimal Viable Product. [online] Steve Blank. Available at: https://steveblank.com/2015/05/06/build-measure-learn-throw-things-against-the-wall-and-see-if-they-work/ [Accessed 25 Dec. 2016]. BotStory. (2016). Top messaging platforms supporting bots. [online] Available at: http://botstory.co/top-platforms-support-bots-chatbots/ [Accessed 28 Dec. 2016]. Boulton, C. (2016). From tacos to HR, chatbots make it personal. [online] CIO. Available at: http://www.cio.com/article/3063051/instant-messaging/from-tacos-to-hr-chatbots- make-it-personal.html [Accessed 2 Jan. 2017]. Business Insider. (2016). Messaging Apps are Now Bigger than Social Networks. [online] Available at: http://www.businessinsider.com/the-messaging-app-report-2015- 11?IR=T&r=US&IR=T [Accessed 1 Jan. 2017].
  36. 36. 36 Chowdhury, G. (2005). Natural language processing. Annual Review of Information Science and Technology, 37(1), pp.51-89. Christensen, C., Hall, T., Dillon, K. and Duncan, D. (2016). Know Your Customers’ “Jobs to Be Done”. [online] Harvard Business Review. Available at: https://hbr.org/2016/09/know- your-customers-jobs-to-be-done [Accessed 26 Dec. 2016]. Dan, A. (2016). Consultants Are Eating The Agencies' Three-Martini Lunch. [online] Forbes.com. Available at: http://www.forbes.com/sites/avidan/2016/04/25/consultants-are- eating-the-agencies-three-martini-lunch/#5ba472b63aba [Accessed 30 Dec. 2016]. Deutscher, M. (2016). FirstJob’s Mya is the latest chatbot that aims to automate recruiting. [online] SiliconANGLE. Available at: http://siliconangle.com/blog/2016/07/11/firstjobs- mya-is-the-latest-chatbot-that-aims-to-automate-recruiting/ [Accessed 2 Jan. 2017]. Dishman, L. (2016). This Chatbot Can Make Sure Your Resume Won't End Up In A Black Hole. [online] Fast Company. Available at: https://www.fastcompany.com/3061677/the- future-of-work/the-chatbot-who-can-make-sure-youll-never-get-radio-silence-after-applyin [Accessed 2 Jan. 2017]. Dunn, J. (2016). We put Siri, Alexa, Google Assistant, and Cortana through a marathon of tests to see who’s winning the virtual assistant race — here’s what we found - Business Insider. [online] Business Insider. Available at: http://www.businessinsider.sg/siri-vs-google-assistant-cortana-alexa-2016-11/?r=US&IR=T#rfBHqO3D8eSOZMdp.97 [Accessed 2 Jan. 2017]. Elliott, S. (2002). Advertising's Big Four: It's Their World Now. [online] Nytimes.com. Available at: http://www.nytimes.com/2002/03/31/business/advertising-s-big-four-it-s-their- world-now.html?pagewanted=all&src=pm [Accessed 20 Dec. 2016]. Ember, S. (2016). Ad Agencies Need Young Talent. Cue the Beanbag Chairs.. [online] Nytimes.com. Available at: http://www.nytimes.com/2016/04/19/business/media/ad-agencies- need-young-talent-cue-the-bean-bag-chairs.html?_r=0 [Accessed 31 Dec. 2016]. Epstein, R., Roberts, G. and Beber, G. (2008). Parsing the turing test. 1st ed. New York: Springer, pp.181-210. Farmer, M. (2015). Madison Avenue Manslaughter. 1st ed. New York: LID Publishing Ltd. Fromowitz, M. (2016). Commoditization: The biggest threat facing ad agencies today. [online] Campaign Asia. Available at: http://www.campaignasia.com/article/commoditization- the-biggest-threat-facing-ad-agencies-today/425997 [Accessed 30 Dec. 2016]. Futurism. (n.d.). The History of Chatbots [INFOGRAPHIC]. [online] Available at: http://futurism.com/images/the-history-of-chatbots-infographic/ [Accessed 1 Jan. 2017]. Gartner.com. (2015). Gartner Says Weak Mobile Customer Service Is Harming Customer Engagement. [online] Available at: http://www.gartner.com/newsroom/id/2956618 [Accessed 1 Jan. 2017]. Grayson, A. and North, G. (2016). How the Advertising Industry is Wasting Talent and What We Can Do About It. [online] Medium. Available at: https://medium.com/@andygrayson/how-the-advertising-industry-is-wasting-talent-and-what-we-can-do-about-it-4824411b9006#.8h9lmf6e3 [Accessed 31 Dec. 2016]. Greenfield, R. (2016). Chatbots Are Your Newest, Dumbest Co-Workers. [online] Bloomberg.com. Available at: https://www.bloomberg.com/news/articles/2016-05-05/chatbots-are- your-newest-dumbest-co-workers [Accessed 2 Jan. 2017].
  37. 37. 37 Griffin, A. (2014). Computer becomes first to pass Turing Test in artificial intelligence. [online] The Independent. Available at: http://www.independent.co.uk/life-style/gadgets-and- tech/computer-becomes-first-to-pass-turing-test-in-artificial-intelligence-milestone-but-academics-warn-9508370.html [Accessed 31 Dec. 2016]. Hobson, N. (2016). Are chatbots in the workplace the entry point to cognitive personal assistants? | NevilleHobson.com. [online] NevilleHobson.com. Available at: http://www.nevillehobson.com/2016/05/10/chatbots-workplace-cognitive/ [Accessed 1 Jan. 2017]. Howard, T. (2016). The Guide To Designing A Magical Chatbot Experience. [online] Chatbots Magazine. Available at: https://chatbotsmagazine.com/the-guide-to-designing-a- magical-chatbot-experience-part-1-efbf32444448#.wyex0oqer [Accessed 3 Jan. 2017]. Init.ai Decoded. (2016). The Product Designer’s Guide to Conversational Commerce. [online] Available at: https://blog.init.ai/the-product-designers-guide-to-conversational- commerce-cbe466753add#.gn5sjy8bk [Accessed 27 Dec. 2016]. Jain, A. (2016). The Flow Framework — How to build a kickass UX for your Chat-bot? (Part — II). [online] ChatterOn. Available at: https://blog.chatteron.io/how-to-build-a-kickass-ux- for-your-chat-bot-part-ii-the-flow-framework-811355905249#.6dlq6d3qw [Accessed 3 Jan. 2017]. Johnson, B. (2015). State of the Agency Market: What You Need to Know. [online] Adage.com. Available at: http://adage.com/article/digital/datacenter-agency-report-2015- charts/298214/ [Accessed 30 Dec. 2016]. Knowledge@Wharton. (2016). The Rise of the Chatbots: Is It Time to Embrace Them? - Knowledge@Wharton. [online] Available at: http://knowledge.wharton.upenn.edu/article/rise-chatbots-time-embrace/ [Accessed 31 Dec. 2016]. Kojouharov, S. (2016). Ultimate Guide to Leveraging NLP & Machine Learning for your Chatbot. [online] Chatbot’s Life. Available at: https://chatbotslife.com/ultimate-guide-to- leveraging-nlp-machine-learning-for-you-chatbot-531ff2dd870c#.zaqk0o35t [Accessed 2 Jan. 2017]. Lamprecht, E. (2016). The Difference Between UX and UI Design-A Layman’s Guide. [online] Blog.careerfoundry.com. Available at: http://blog.careerfoundry.com/ui-design/the- difference-between-ux-and-ui-design-a-laymans-guide/ [Accessed 4 Jan. 2017]. Levinson, R. (2016). The tools every bot creator must know. [online] Chatbots Magazine. Available at: https://chatbotsmagazine.com/the-tools-every-bot-creator-must-know- c0e9dd685094#.v6h0lbonf [Accessed 28 Dec. 2016]. Lo, M. (2014). UI, UX: Who Does What? A Designer's Guide To The Tech Industry. [online] Co.Design. Available at: https://www.fastcodesign.com/3032719/ui-ux-who-does-what-a- designers-guide-to-the-tech-industry [Accessed 4 Jan. 2017]. Loebner, H. (n.d.a). In Response. [online] Loebner.net. Available at: http://loebner.net/Prizef/In-response.html [Accessed 31 Dec. 2016]. Loebner.net. (n.d.b). Home Page of the Loebner Prize in Artificial Intelligence. [online] Available at: http://www.loebner.net/Prizef/loebner-prize.html [Accessed 31 Dec. 2016]. Maass, D. (2014). Answers and Questions About Military, Law Enforcement, and Intelligence Agency Chatbots. [online] Electronic Frontier Foundation. Available at: https://www.eff.org/deeplinks/2014/04/answers-questions-about-military-law-enforcement-and-intelligence-agency-chatbots [Accessed 2 Jan. 2017]. Marcus, D. (2016). Messenger Platform at F8. [online] Newsroom.fb.com. Available at: http://newsroom.fb.com/news/2016/04/messenger-platform-at-f8/ [Accessed 1 Jan. 2017].
  38. 38. 38 Marr, B. (2016). What Is The Difference Between Artificial Intelligence And Machine Learning?. [online] Forbes.com. Available at: http://www.forbes.com/sites/bernardmarr/2016/12/06/what-is-the-difference-between-artificial-intelligence-and-machine-learning/3/#657340f11a30 [Accessed 1 Jan. 2017]. Marshall, J. and Alpert, L. (2016). Publishers Take On Ad-Agency Roles With Branded Content. [online] WSJ. Available at: http://www.wsj.com/articles/publishers-take-on-ad- agency-roles-with-branded-content-1481457605 [Accessed 30 Dec. 2016]. Maurya, A. (2012). Running Lean: Iterate from a Plan A to a Plan that Works. 2nd ed. Sebastopol, CA: O'Reilly. McNeale, M. and Newyear, D. (2013). Introducing Chatbots in Libraries. Library Technology Reports, 49(8), pp.5 - 10. Medium. (2016). 聊天機器人市場版圖 — Chatbots Landscape. [online] Available at: https://medium.com/@compose.ai/%E8%81%8A%E5%A4%A9%E6%A9%9F%E5%99%A8%E4%BA%BA%E5%B8%82%E5%A0%B4%E7%89%88%E5%9C%96-chatbots- landscape-f89302206875#.wk2de23dy [Accessed 2 Jan. 2017]. Meeker, M. (2016). 2016 Internet Trends Report. [online] Kpcb.com. Available at: http://www.kpcb.com/internet-trends [Accessed 1 Jan. 2017]. Mindshare, Goldsmiths University of London, (2016). Humanity in the Machine. [online] Available at: http://www.mindshareworld.com/sites/default/files/MINDSHARE_HUDDLE_HUMANITY_MACHINE_2016_0.pdf [Accessed 1 Jan. 2017]. Morrison, M. (2016). Sprint Names Exec Creative Director to Run New In-House Agency. [online] Adage.com. Available at: http://adage.com/article/agency-news/sprint-staffs-house- agency/304045/ [Accessed 30 Dec. 2016]. Mumbrella Asia. (2016). Lou Dela Pena handed broader role leading Publicis Communications in Singapore - Mumbrella Asia. [online] Available at: http://www.mumbrella.asia/2016/01/lou-dela-pena-handed-broader-role-leading-publicis-communications-in-singapore/ [Accessed 20 Dec. 2016]. myclever Agency, (2016). Chatbots: A Consumer Research Study. [online] London. Available at: http://www.slideshare.net/mycleveragency?utm_campaign=profiletracking&utm_medium=sssite&utm_source=ssslideview [Accessed 1 Jan. 2017]. Newman, J. (2016). How The New, Improved Chatbots Rewrite 50 Years Of Bot History. [online] Fast Company. Available at: https://www.fastcompany.com/3059439/why-the-new- chatbot-invasion-is-so-different-from-its-predecessors [Accessed 31 Dec. 2016]. Newton, C. (2016). Can AI fix education? We asked Bill Gates. [online] The Verge. Available at: http://www.theverge.com/2016/4/25/11492102/bill-gates-interview-education- software-artificial-intelligence [Accessed 2 Jan. 2017]. Nextit.com. (n.d.). SGT STAR helps potential recruits learn about Army life. [online] Available at: http://www.nextit.com/work/army.php [Accessed 2 Jan. 2017]. Nickerson, R. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), pp.175-220. Oetting, J. (2015). The Biggest Threats to the Agency Model. [online] Blog.hubspot.com. Available at: https://blog.hubspot.com/agency/threats-agency- model#sm.000q8a2qc1192eo6ypt159w7mslve [Accessed 30 Dec. 2016].
  39. 39. 39 Olson, P. (2016). Duolingo's New Chat Bots Can Teach You French, German And Spanish. [online] Forbes.com. Available at: http://www.forbes.com/sites/parmyolson/2016/10/07/duolingo-chat-bots-artificial-intelligence/#2d80c93766e5 [Accessed 2 Jan. 2017]. Ovchinnikova, E. (2012). Integration of world knowledge for natural language understanding. 1st ed. Amsterdam: Atlantis Press, p.15. Pandey, R. (2016). Global ad spend: Trends for 2016 and 2017. [online] Marketing Interactive. Available at: http://www.marketing-interactive.com/global-ad-spend-trends-2016- 2017/ [Accessed 30 Dec. 2016]. Pearce, K. (2013). Ask Ivy. [online] Jobs@Intel Blog. Available at: http://blogs.intel.com/jobs/2013/05/16/ask-ivy/ [Accessed 2 Jan. 2017]. Perez-Marin, D. and Pascual-Nieto, I. (2011). Conversational Agents and Natural Language Interaction. 1st ed. Hershey, PA: Information Science Reference, pp.1-22, 107-127, 358- 378. Publicis Groupe, (2016a). Publicis Groupe Announces Important Nominations and its Transformation Plan. [online] Available at: http://livredor.publicis.com/notesInternes/2015/021215_EN_PublicisGroupe_Transformation_DEF.pdf [Accessed 20 Dec. 2016]. Publicis Groupe, (2016b). Third Quarter 2016 Revenue. [online] Paris: Publicis Groupe. Available at: http://www.publicisgroupe.com/media/display/id/8072.pdf [Accessed 31 Dec. 2016]. Publicis Groupe, (2016c). 2015 Registration Document - Annual Financial Report. [online] Paris: Publicis Groupe, pp.90 - 95. Available at: http://www.publicisgroupe.com/documents/document-de-reference-2015-en.pdf [Accessed 31 Dec. 2016]. Richman, D. (2016). Microsoft Azure offers ‘bots as a service’ as demand rises for automated online interactions. [online] GeekWire. Available at: http://www.geekwire.com/2016/microsoft-azure-offers-bots-service-demand-rises-automated-online-interactions/ [Accessed 5 Jan. 2017]. Riel, J. (2016). How Chatbots Can Help With Learning. [online] College of Education | University of Illinois Chicago. Available at: https://education.uic.edu/academics- admissions/student-life/how-chatbots-can-help-learning [Accessed 2 Jan. 2017]. Ries, E. (2011). The Lean Startup. 1st ed. London: Penguin Group, pp.56-78. Rosenberg, S. (2016). New Messenger Platform Features: Link Ads to Messenger, Enhanced Mobile Websites, Payments and More. [online] Facebook for Developers. Available at: https://developers.facebook.com/blog/post/2016/09/12/new-messenger-features-payments-ads-enhanced-mobile-websites/ [Accessed 1 Jan. 2017]. Roxburgh, H. (2016). Why big consultancies buy design agencies. [online] Campaign Asia. Available at: http://www.campaignasia.com/article/why-big-consultancies-buy-design- agencies/407973 [Accessed 30 Dec. 2016]. Sas.com. (n.d.). Machine Learning: What it is and why it matters. [online] Available at: http://www.sas.com/en_us/insights/analytics/machine-learning.html [Accessed 1 Jan. 2017]. Schaefer, M. (2015). 6 Reasons Marketing Is Moving In-House. [online] Harvard Business Review. Available at: https://hbr.org/2015/07/6-reasons-marketing-is-moving-in-house [Accessed 30 Dec. 2016]. Schlicht, M. (2016). The Complete Beginner’s Guide To Chatbots. [online] Chatbots Magazine. Available at: https://chatbotsmagazine.com/the-complete-beginner-s-guide-to- chatbots-8280b7b906ca#.dxwvm1po5 [Accessed 31 Dec. 2016].

×