Presentation @ PyTorch London, November 2019

1. Using language models
to supercharge Monzo’s
customer support
@neal_lathia
November 5, 2019

3. See this post by @jackkleeman
https://monzo.com/blog/we-built-network-isolation-for-1-500-services

4. Many different
machine learning
problems!

5. Enabling swift, delightful
customer service.

6. 1. Helping customers find the right
answers to their queries in the app.
//
2. Helping agents to diagnose and
response to customer queries swiftly.

8. Two approaches we
are working on

9. ⬅ Prompt for information
⬅ Recommendations (search
results)
⬅ Self-selected chat closure
⬅ The chat first turn (this could
be more than one message)

10. Formulation
Article recommendations as a relevance
problem: can we improve our search
results?

11. General Approach
The customer query is turned into a
numerical representation (an embedding).
We also keep all of the latest help article
embeddings.
The top recommendations are the help
articles that have the most similar
embeddings (Cosine distance ranking).

12. You can change your
card PIN at any large
bank (HSBC,
Barclays, etc.) ATM
in the UK by
selecting PIN
services ☺
You can see and
download your bank
statements through
the Monzo app.
How do I change
my PIN?
Question + Answer - Answer

13. Baseline
Train an encoder-decoder model from
scratch, using Monzo chat data only
Challenger
Use a pre-trained BERT encoder, and
finetune it using Monzo Chat data
Result!
Challenger ⬆ self-service rate by ~9%

14. Experimental
classification
approach

15. From our ongoing analysis, we can identify
topics that are:
Eligible for self-service (e.g., PIN reset)
Predictable (in our chat data)
Valuable for customer service (i.e.,
accounts for a high volume of their work)
... perhaps we could approach this as a
classification problem instead?

16. We were looking for an approach that is:
🎯 Accurate
We were looking for a way where:
🚢 We could ship quickly to get some customer
feedback
🏦 Use all the relevant state that is stored across
many backend services

17. Approach
Finetune a single binary classifier
about a topic: is the customer asking
about needing a replacement card?
If they are (& a card hasn’t already been
ordered for them), then give them the
answer directly!

18. Offline validation
Precision
Out-of-sample
precision
Card replacement 0.86 0.69
Card not arrived 0.93 0.60
Update details 0.93 0.52 😫
... ... ...
We deployed a number of these models in shadow
prediction mode. This allows it to make predictions on
live data, but it’s not sending customers any answers.

19. What the 🤬?
The main culprit was poorly tagged
conversations. But the definition of
“correctly” tagged was subjective!

20. Large set of noisy tags
Small set of re-labelled
examples
Hi there! How can I transfer money into a savings pot?

21. F1 Score
Fine tune on tags 69
Fine tune on
re-labelled
examples
78
“Ultra” fine-tune on
both
85
Example result: Update details (48% of
tagged chats labels changed when
re-labelled)

22. Conclusion - how do
we enable this work?

23. Orchestrator
Classifier for A
Classifier for B
Classifier for C

24. Orchestrator
Classifier for A
Classifier for B
Classifier for C
0.0%
8.0%
98.0%
Topic: C

25. Orchestrator
Classifier for A
Classifier for B
Classifier for C
Winner!
Rules that determine
eligibility for topic C

26. Orchestrator
Classifier for A
Classifier for B
Classifier for C
Winner!
Rules that determine
eligibility for topic C
Classifier for D

27. 1. Helping customers find the right
answers to their queries (in the app)
//
2. Helping agents to diagnose and
response to customer queries swiftly.

28. Thanks!
@neal_lathia
October 8, 2019