Download to read offline
![[DSC Europe 24] Guilherme Diaz-Berrio - The (mis)use of GenAI in Data Analytics](https://cdn.slidesharecdn.com/ss_thumbnails/guilhermediaz-berrio-250212202643-d5c8e538-thumbnail.jpg?width=640&height=640&fit=bounds)
EssentialPIM Pro Business Free Download
- 1. Where there’s awill, there’s a way The mad genius of using LLMs as classifiers Katherine Munro, DSC Belgrade 2024
- 2. 2 About Me Data Scientist, ComputationalLinguist, Conversational AI Engineer Catch me talking about all things data, AI, NLP, and innovation, here… linkedin.com/in/katherine- munro katherine-munro.com Or else, just find me in the mountains … Katherine Munro, DSC Belgrade, November 2024
- 3. What we’ll talkabout: • LLMs as classifiers:A weird idea you should probably try anyway • Intent Detection: A real-world classification use case • Possible architectures and techniques • Generalizing to other classification tasks Katherine Munro, DSC Belgrade, November 2024
- 4. 4 LLMs as classifiersis a weird idea… A Machine Learning (ML) technique where a model must classify an input into one of a set of possible classes, denoted by a single label. For example: • Sentiment Detection • Email Triage Complex algorithms able to solve diverse tasks and respond with diverse, long-form outputs. Classification: Large Language Models: Typically, classification should be: • Accurate • Consistent • Interpretable • Fast LLMs are: • Slow • Random • Not at all interpretable Katherine Munro, DSC Belgrade, November 2024
- 5. Depending on themethod used, LLM classifiers offer the following benefits: • Useful for prototyping. • No training required. • Scalability and flexibility when adding new classes. • An existing body of data sets, evaluation metrics and best practices. • Ability to handle diverse, inconsistent, multi-modal data. • Easily transferable to multiple languages. 5 … but you might want to do classification with LLMs anyway Katherine Munro, DSC Belgrade, November 2024
- 6. Introduction to Intent Detection Areal-world classification use case Katherine Munro, DSC Belgrade, November 2024
- 7. When customers call,we need to quickly and accurately route them to a customer service team or a self-service use case. This requires detecting which product or service they’re calling about, and what need they’re trying to solve. This is a classification task. 7 Intent detection for a conversationalAI agent Katherine Munro, DSC Belgrade, November 2024
- 8. Variability in howcustomers express themselves is confusing, even for humans. Meaning is built up across multiple turns of dialogue. Context is key: Which services does the customer have? What’s the status of those services? Customers don’t always know what they want or need. Misalignment between how customers think about problems and how we’re set up to help them. Customers can have multiple intents. Data “noise”, e.g. speech transcription errors for our voicebot. 8 Why intent detection is hard… Helpful resource: “HowYour Digital Personal Assistant Understands WhatYou Want (And Gets it Done)" Katherine Munro, DSC Belgrade, November 2024
- 9. Despite the challenges,there are also benefits to this approach. Intent-based logic: • helps us simplify natural language to make it workable • helps us plan quality customer experiences using conversation design • makes our system more deterministic, interpretable and testable • can reduce hallucinations and improve efficiency by eradicating unnecessary function or API calls • can be practical and logical for other parts of the business, e.g. for reporting and planning employee training 9 … But we do it anyway Helpful resource: “Why you still need NLP skills in the ‘age of ChatGPT’" Katherine Munro, DSC Belgrade, November 2024
- 10. If intents areso tricky, and LLMs so great, why do intent detection at all? Why not use an LLM end to end? It’s possible. But: • Routine use cases don’t need an LLM’s creativity or spontaneity. • LLMs can be a better “front door”, whenever you have good routing logic but poor intent detection accuracy. • You might not trust an LLM with your most valuable interactions – customer contacts. • Converting an open problem to a closed one, and breaking it down into stages, are already prompting best practices. • You can still use LLMs for other use cases, e.g RAG and chit-chat. 10 Couldn’t we use LLMs for the lot? Katherine Munro, DSC Belgrade, November 2024
- 11.
- 12. In a classicNLU bot, a trained ML model performs the intent (and possibly) topic detection. Business logic, encoded as rules, uses these predictions to define a final routing. LLMs can be used not for detection, but for rephrasing system utterances. 12 “The classic” Katherine Munro, DSC Belgrade, November 2024 Pros: ML models can be simple, small, fast, interpretable, and highly specified. Evaluation is concrete and can be automated. Cons: Adding new prediction classes and implementing the business logic is not very scalable. Real-world example: Many, many Conversational AI systems that exist today.
- 13. This approach stilluses an ML model for the initial classification, but defers to an LLM when the prediction uncertainty is high. 13 “The hybrid” Helpful resource: Vux Podcast: The AI chatbot serving 10m customers a year Katherine Munro, DSC Belgrade, November 2024 Pros: All the benefits of ML models, plus LLM strengths, somewhat cheap. Cons: System becomes more complex to deploy and maintain. LLMs add expense, latency, opacity and unpredictability. Real-world example: Lufthansa Group
- 14. Another hybrid approach:AnNLU model retrieves the top N most likely intents. These are injected into a prompt and given to an LLM to make the final classification. 14 “The filter” Helpful resource: Benchmarking hybrid LLM classification systems,Voiceflow Katherine Munro, DSC Belgrade, November 2024 Pros: Gives a chance to recover from poor ML accuracy or model drift. Cons: Same as before. Real-world example:Voiceflow
- 15. AKA “few-shot prompting”:have your bot make a prediction based on the descriptions, and a few examples, of the possible intents. 15 “The fast learner” Katherine Munro, DSC Belgrade, November 2024 Pros: Uses LLMs’ out-of-the-box capabilities, making implementing new cases much more scalable. Cons: All the issues LLMs bring, especially costs, security and latency challenges. Real-world example: Rasa
- 16. With few-shot inference,as you add more use cases, the prompt explodes, causing latency and accuracy issues. More classes can also lead to lower confidence and more fallbacks. In an embedding approach: • you embed the intent labels and descriptions • retrieve the ones that are most similar to the (embedded) user utterance • and inject only those into the LLM prompt. 16 “The embedder” Katherine Munro, DSC Belgrade, November 2024 Pros: Tackles latency and accuracy problems and adds some interpretability. Cons: Dissimilarities between customers’ spoken style and developer descriptions can make matching difficult, and thus impact accuracy. Real-world example: Rasa
- 17. LLM next tokenprediction is essentially a classification task, where the possible labels are its token vocabulary. To fine-tune an LLM for classification: • Attach a custom head to the model, e.g. a Gradient Boosted tree or logistic regression • Fine tune it to map the logit distribution of the whole vocabulary to just the output labels you want. • Could also fine-tune the LLM combined with this head, e.g. LORA training for the LLM component 17 “The tuner” Katherine Munro, DSC Belgrade, November 2024 Pros: Could provide more control, interpretability and accuracy. Cons: All the complexity that comes with adding an ML model to your stack. LLMs for Classification Tasks with LLM Studio Helpful resource: Mastering Classification & Regression with LLMs: Insights from Kaggle Competitions
- 18. Before: The Classic:AnNLU based bot, featuring logical rules based on predictions by ML classifiers for intent and topic. Right Now: The Fast Learner: Few-shot inference is working well, especially for prototyping. In the future, we’d like to try: • Approaches which reduce the number of possible labels for the LLM, e.g. The Embedder • Multi-modal models: to (hopefully) make our system simpler, faster, and less sensitive to ASR issues 18 What we’ve tried so far Helpful resource: 5 Ways to Optimize your Prompts for LLM Intent Classification Katherine Munro, DSC Belgrade, November 2024
- 19. Katherine Munro, DSC Belgrade, November 2024 What’s in itfor you? Generalizing to other classification tasks
- 20. These architectures canbe applied to any kind of labelled classification problem. Start simple and try it out!: • Grab some production data • Label it: It’s worth the effort! • Establish your baselines: simple majority, and existing classifier accuracy • Try few-shot inference, either via an API or direct in the UI • Hybrid Approach: Test only the samples where the current classifiers fail; add the results • Fast-learner Approach: Compare all samples against existing classifier predictions If results are promising, you can think about testing more complex approaches, i.e. filtering possible intents using ML confidence scores (The Filter) or embeddings (The Embedder), or fine-tuning a classifier component (TheTuner). 20 Putting it into practice Katherine Munro, DSC Belgrade, November 2024
- 21. Prompting “best practices”can be unexpected and model-dependent. Experiment a lot, and document what works best for your problem. Include domain experts, e.g. conversation designers or call-centre agents, when designing prompts and exploring model outputs. Remember data protection laws when testing a public LLM! Situate your problem in a business context. For example: • What are the business impacts of different kinds of misclassifications? • Is your data telling you the full story? Get clever about how to measure progress. 21 A FewTopTips Katherine Munro, DSC Belgrade, November 2024 Helpful resource: No Baseline? No Benchmarks? No Biggie! An Experimental Approach to Agile Chatbot Development
- 22.