⬅ Prompt for information
⬅ Recommendations (search
⬅ Self-selected chat closure
⬅ The chat ﬁrst turn (this could
be more than one message)
Article recommendations as a relevance
problem: can we improve our search
The customer query is turned into a
numerical representation (an embedding).
We also keep all of the latest help article
The top recommendations are the help
articles that have the most similar
embeddings (Cosine distance ranking).
You can change your
card PIN at any large
Barclays, etc.) ATM
in the UK by
You can see and
download your bank
the Monzo app.
How do I change
Question + Answer - Answer
Train an encoder-decoder model from
scratch, using Monzo chat data only
Use a pre-trained BERT encoder, and
ﬁnetune it using Monzo Chat data
Challenger ⬆ self-service rate by ~9%
From our ongoing analysis, we can identify
topics that are:
Eligible for self-service (e.g., PIN reset)
Predictable (in our chat data)
Valuable for customer service (i.e.,
accounts for a high volume of their work)
... perhaps we could approach this as a
classiﬁcation problem instead?
We were looking for an approach that is:
We were looking for a way where:
🚢 We could ship quickly to get some customer
🏦 Use all the relevant state that is stored across
many backend services
Finetune a single binary classiﬁer
about a topic: is the customer asking
about needing a replacement card?
If they are (& a card hasn’t already been
ordered for them), then give them the
Card replacement 0.86 0.69
Card not arrived 0.93 0.60
Update details 0.93 0.52 😫
... ... ...
We deployed a number of these models in shadow
prediction mode. This allows it to make predictions on
live data, but it’s not sending customers any answers.
What the 🤬?
The main culprit was poorly tagged
conversations. But the deﬁnition of
“correctly” tagged was subjective!
Large set of noisy tags
Small set of re-labelled
Hi there! How can I transfer money into a savings pot?
Fine tune on tags 69
Fine tune on
“Ultra” fine-tune on
Example result: Update details (48% of
tagged chats labels changed when