Download as PDF, PPTX
Uploaded byNeal Lathia
Using language models to supercharge Monzo’s customer support
1. Monzo is working on using language models to improve customer support by helping customers find answers to their queries in the app and helping agents swiftly respond to customer queries. 2. They are taking two approaches - using models to recommend relevant search results and help articles, and using classification models to directly answer questions about common topics like changing a PIN number or replacing a card. 3. Their experiments found that fine-tuning a pre-trained BERT model on their chat data improved the self-service rate for customers compared to training a model from scratch. However, noisy and subjective tagging of the training data was an issue, so they achieved better results by re-labeling examples.
More Related Content
Similar to Using language models to supercharge Monzo’s customer support
Using language models to supercharge Monzo’s customer support
- 1. Using language models tosupercharge Monzo’s customer support @neal_lathia November 5, 2019
- 3. See this postby @jackkleeman https://monzo.com/blog/we-built-network-isolation-for-1-500-services
- 4.
- 5.
- 6. 1. Helping customersfind the right answers to their queries in the app. // 2. Helping agents to diagnose and response to customer queries swiftly.
- 8. Two approaches we areworking on
- 9. ⬅ Prompt forinformation ⬅ Recommendations (search results) ⬅ Self-selected chat closure ⬅ The chat first turn (this could be more than one message)
- 10. Formulation Article recommendations asa relevance problem: can we improve our search results?
- 11. General Approach The customerquery is turned into a numerical representation (an embedding). We also keep all of the latest help article embeddings. The top recommendations are the help articles that have the most similar embeddings (Cosine distance ranking).
- 12. You can changeyour card PIN at any large bank (HSBC, Barclays, etc.) ATM in the UK by selecting PIN services ☺ You can see and download your bank statements through the Monzo app. How do I change my PIN? Question + Answer - Answer
- 13. Baseline Train an encoder-decodermodel from scratch, using Monzo chat data only Challenger Use a pre-trained BERT encoder, and finetune it using Monzo Chat data Result! Challenger ⬆ self-service rate by ~9%
- 14.
- 15. From our ongoinganalysis, we can identify topics that are: Eligible for self-service (e.g., PIN reset) Predictable (in our chat data) Valuable for customer service (i.e., accounts for a high volume of their work) ... perhaps we could approach this as a classification problem instead?
- 16. We were lookingfor an approach that is: 🎯 Accurate We were looking for a way where: 🚢 We could ship quickly to get some customer feedback 🏦 Use all the relevant state that is stored across many backend services
- 17. Approach Finetune a singlebinary classifier about a topic: is the customer asking about needing a replacement card? If they are (& a card hasn’t already been ordered for them), then give them the answer directly!
- 18. Offline validation Precision Out-of-sample precision Card replacement0.86 0.69 Card not arrived 0.93 0.60 Update details 0.93 0.52 😫 ... ... ... We deployed a number of these models in shadow prediction mode. This allows it to make predictions on live data, but it’s not sending customers any answers.
- 19. What the 🤬? Themain culprit was poorly tagged conversations. But the definition of “correctly” tagged was subjective!
- 20. Large set ofnoisy tags Small set of re-labelled examples Hi there! How can I transfer money into a savings pot?
- 21. F1 Score Fine tuneon tags 69 Fine tune on re-labelled examples 78 “Ultra” fine-tune on both 85 Example result: Update details (48% of tagged chats labels changed when re-labelled)
- 22. Conclusion - howdo we enable this work?
- 23. Orchestrator Classifier for A Classifierfor B Classifier for C
- 24. Orchestrator Classifier for A Classifierfor B Classifier for C 0.0% 8.0% 98.0% Topic: C
- 25. Orchestrator Classifier for A Classifierfor B Classifier for C Winner! Rules that determine eligibility for topic C
- 26. Orchestrator Classifier for A Classifierfor B Classifier for C Winner! Rules that determine eligibility for topic C Classifier for D
- 27. 1. Helping customersfind the right answers to their queries (in the app) // 2. Helping agents to diagnose and response to customer queries swiftly.
- 28.