As educators, we have all encountered the scenario of receiving repeated emails from students (or a posts on a discussion board) requesting information that is contained within the subject outline, is on UTSOnline or is available on the UTS website. In an introductory undergraduate accounting subject, we designed an automated chatbot to attempt to relieve some of the administrative load of subject coordinators. In this presentation we will discuss the process and challenges related to creating a chatbot, preliminary data on usage by students and future implementation plans.

1. UTS CRICOS 00099F
Creating a chatbot to reduce
the administrative load and
emotional labour of subject
coordinators
First and Further Year Experience Grant
Dr Amanda White, Dr Nelson Ma, Dr James Wakefield, Atieh Fallahi
UTS Business School
UTS T&L Forum 2019
this information is
already on
UTSOnline

2. The challenge for coordinators
Where
important
information for
a subject is
stored
What students
ACTUALLY do
when they need to
find out important
subject
information
UTSOnline
Subject
outline
Assessment
guide
Post on the
discussion board
Email the coordinator
Ask a friend
Look on UTSOnline
Read
the
outline

3. For Introductory first year accounting...
The (additional) challenge – Disseminating a consistent message
Large student cohorts (600 – 1,500 students per session)
Multiple lectures per week
3 contact hours per week – 1.5 hour lecture and 1.5 hour tutorial
Tutorials typically covered by honours/PhD students and casual staff
Non-mandatory attendance
Reliance on LMS to contact students
The result? (too) many student emails and duplicate discussion board posts

4. What is the impact on staff? Emotional labour
• Emotional labour = induce
or suppress feeling in order
to sustain the outward
countenance that produces
the proper state of mind in
others
Hochschild (2012), The
Managed Heart,
https://www.ucpress.edu/
ebook.php?isbn=978052
0951853
https://www.starwars.com/databank/yoda
Repetitive requests
lead to frustration.
Frustration leads to a
decline in the
relationship between
student and teacher.
That decline affects
the student
experience and
performance.

5. Our attempt at a solution
Inspired by a workshop by these
2 visitors to UTS
Jonathan Grudin
Principal Researcher -
Microsoft Research
Dr David Kellerman
Senior Lecturer - UNSW
What if we could build a
CHAT BOT to answer
these questions?

6. UTS CRICOS 00099F
Meet Penny
The trial
Subject: 22107: Accounting for Business Decisions A (Spring 2019)
Dates: 4th-30th October 2019
Cohort: approx. 550 students
Provider: UTS ChatBot Team

7. Promotion and access
Promotion: UTS Online
Announcement
Access: Hyperlink on UTS Online Page

8. UTS CRICOS 00099F
The Results
Key points:
• 72 Unique Student Logons.
• 98 Student Questions and Requests (Intent).
• 42% successfully answered.
• 52% involved referrals to other ChatBots.
• Topics were wide ranging but more focused on the final exam.

9. UTS CRICOS 00099F
Question to Outcome
OutcomeResponseQuestion
Student
Question/Request
(n=98)
ChatBot Answer
(48%)
Successful (64%)
Unsuccessful (36%)
Referral to Other
ChatBot (52%)
Successful (24%)
Unsuccessful (76%)

10. Descriptive Statistics – Topics
Table 1: Student Questions, Requests and Statements
Panel B: Student Question/Request Topics (n=98)
Final Exam (timing, format, topics covered etc.)
Content (e.g. "what is accounts receivable?")
Grades (timing of release etc.)
Other Assessment (class participation, mid-session and screencast)
Illness (inability to attend exam etc.)
Other (e.g. "when is the lecture?" and "Where is Morpheus?")
Total Student Questions/Responses

11. Descriptive Statistics – Topics
Table 1: Student Questions, Requests and Statements
Total
Panel B: Student Question/Request Topics (n=98) n %
Final Exam (timing, format, topics covered etc.) 43 44%
Content (e.g. "what is accounts receivable?") 11 11%
Grades (timing of release etc.) 10 10%
Other Assessment (class participation, mid-session and screencast) 13 13%
Illness (inability to attend exam etc.) 6 6%
Other (e.g. "when is the lecture?" and "Where is Morpheus?") 15 15%
Total Student Questions/Responses 98 100%

12. Descriptive Statistics – Topics
Table 1: Student Questions, Requests and Statements
Total Successful
Panel B: Student Question/Request Topics (n=98) n % n %
Final Exam (timing, format, topics covered etc.) 43 44% 21 49%
Content (e.g. "what is accounts receivable?") 11 11% 1 9%
Grades (timing of release etc.) 10 10% 6 60%
Other Assessment (class participation, mid-session and screencast) 13 13% 9 69%
Illness (inability to attend exam etc.) 6 6% 2 33%
Other (e.g. "when is the lecture?" and "Where is Morpheus?") 15 15% 2 13%
Total Student Questions/Responses 98 100% 41 42%

13. Teething issues
• Slight variations in wording / single word requests:
"Screencast"
"Screen Cast"
• Reliance on other chatbots:
"I missed the exam" ---> Student Centre Services
"Illness" ---> Staff Leave
• Offensive presumptions:
Student Question: "I love Justin Bieber"
ChatBot Response: "UTS offers a confidential counselling service..."

14. Going Forward....
• Disconnecting the ChatBot referrral system.
• Broadening the question/intent pool.
• Second Trial in 22207: Accounting for Business Decisions B in Summer 2019.
o November 2019 – February 2020 (Full Session)
o Cohort: Approx. 300 students

15. UTS CRICOS 00099F
Thank you
Questions?