Monzo Bank developed a chatbot to help customers with common questions through self-service. Their first iteration involved adding an article search to chat. This showed promise, but still required effort from customers. Their second iteration directly answered questions about predictable topics using classifiers, like card replacements. They tested topics in "shadow mode" and deployed those that predicted accurately. They designed the system as independent classifiers for each topic that were orchestrated, allowing easier collaboration and expansion.

1. Using machine learning
for customer service
@neal_lathia
January 2021 - Data Talks Club

2. Monzo is a UK bank

3. Rewind to 2019, when we
started a project on chat bots
● What were we trying to achieve? How can
we measure this?
● How can we design, build, and iterate on a
bot? How can we demonstrate that it is
working?
● How did we ship this system? How did we
trade off between system design and
machine learning?

4. Framing the problem

5. What do customers need and
how do we help them?
Customer needs
information
Customer needs an
action
Self-Service is available Service via humans
Does my card work
abroad?
Can I have a bank
statement?
What is the status of
the cheque that I sent
you?
I don’t recognise this
transaction, can you
investigate?

6. Self-service is the fastest way to get help for a subset of
queries and enables agents to focus on where they are
really needed
Customer needs
information
Customer needs an
action
Self-Service is available Service via humans
Does my card work
abroad?
Can I have a bank
statement?
What is the status of
the cheque that I sent
you?
I don’t recognise this
transaction, can you
investigate?

7. How can we get a sense of
the size of these problems?
Several different metrics available:
● How many customers are getting in touch?
(Overall contact rate)
● How many customers are getting in touch
about specific topics? (Contact rate by
conversation tag)
● How many customers are using
self-service product features compared to
those getting in touch about the problem?

8. Iteration #1:
Search for article
recommendations

9. Our app already had a search
feature
⬅ Search query
⬅ Used to be an entry point to
chat
⬅ Help article search results

10. ⬅ Prompt for information
⬅ Recommendations (search
results)
⬅ Self-selected chat closure
⬅ The chat first turn (this could
be more than one message)
Our first experiment: add search into chat

11. Why take this approach?
● The product team could focus on adding all of the
components to make this work
○ Typing detection
○ New types of chat bubbles
○ Auto-close if the customer abandons the chat for a
long time
● The analysts could set up the new metrics for this
system
○ Self-selected (& auto) chat close rate
○ Customers proceeding through to chat
● We didn’t need to ship any new ML systems
○ We ran our first BERT experiment, which showed a
+9% increase in self-service rate compared to our
in-house model

12. We ran our first BERT
experiment
● We didn’t need to ship any new ML systems
● The product team could focus on adding all of the
components to make this work
○ Typing detection
○ New types of chat bubbles
○ Auto-close if the customer abandons the chat for a
long time
● The analysts could set up the new metrics for this
system
○ Self-selected (& auto) chat close rate
○ Customers proceeding through to chat

13. Iteration #2:
Giving customers
answers for some
queries

14. Why change the approach?
Article recommendation is working okay, but it still requires
customers to put in the effort to find out if the suggestions
are relevant
We could identify specific topics that our
customers talked to us about that were:
● Valuable - we get a lot of queries about the
topic
● Eligible for self service - there is a simple
and direct call to action
● Predictable - we could train classifiers to
detect the topic!

15. ⬅ Prompt for information
⬅ Predefined answer with a
direct call to action
⬅ Self-selected chat closure
⬅ The chat first turn (this could
be more than one message)

16. How can we test this?
We did not want to build out an entire system (weeks of
work) before validating whether this could action work. So
we started with one topic.
● Fine-tune a BERT model based on tagged
conversations
● Deploy this model in shadow mode to
monitor the prediction quality
● If the accuracy looks good, deploy this topic!
Combine giving answers (for this topic) with
giving article suggestions (for everything
else).

17. Offline validation
Precision
Out-of-sample
precision
Card replacement 0.86 0.69
Card not arrived 0.93 0.60
Update details 0.93 0.52 😫
... ... ...
We deployed a number of these models in shadow
prediction mode. This allows it to make predictions on
live data, but it’s not sending customers any answers.

18. Large set of noisy tags
Small set of re-labelled
examples
Hi there! How can I transfer money into a savings pot?

19. System design

20. Why consider this angle?
We wanted to collaborate!
● Having one huge multi-class model means
retraining everything when we want to add
a new topic
● It is very difficult for two Machine Learning
Scientists to work on one model
collaboratively

21. A bank built with thousands
of microservices
https://qconlondon.com/london2020/presentation/modern-banking-1500-microservices

22. Orchestrator
Classifier for A
Classifier for B
Classifier for C
Winner!

23. Orchestrator
Classifier for A
Classifier for B
Classifier for C
Winner!
Rules that determine
eligibility for topic C
Classifier for D

24. Thanks!
@neal_lathia

25. Some extra writing about
these topics
● Customer service is full of machine
learning problems
● Shadow mode deployments
● Machine learning, faster