Download to read offline
Uploaded byZilliz
Constrained Sampling from Large Language Models: Producing Structured Output
The document outlines several technical talks related to unstructured data and the challenges of semantic search, emphasizing the importance of deep learning similarity search and the role of vector databases like Milvus in addressing these issues. It discusses how Milvus can facilitate operations in generative AI projects with easy setup and integration capabilities, and features a section on constrained sampling from large language models for producing structured output. The document also invites readers to join upcoming meetups focused on related topics.
More Related Content
Similar to Constrained Sampling from Large Language Models: Producing Structured Output
Constrained Sampling from Large Language Models: Producing Structured Output
- 1. 1 | ©Copyright 2024 Zilliz 1 1 | © Copyright 10/22/23 Zilliz 1 | © Copyright 2024 Zilliz Stefan Webb Developer Advocate, Zilliz stefan.webb@zilliz.com https://www.linkedin.com/in/stefan-webb https://x.com/stefan_webb Unstructured Data Meetup | Host
- 2. 2 | ©Copyright 2024 Zilliz 2 2 | © Copyright 10/22/23 Zilliz 2 | © Copyright 2024 Zilliz Welcome Speakers Constrained Sampling from LLMs: Producing Structured Output Infinite World Generation with Outlines BAML Cheaper, faster, and more accurate function calling - without any fine-tuning TECH TALK 1 TECH TALK 2 TECH TALK 3 Stefan Webb DevRel, Zilliz Cameron Pfiffer Developer Relations Engineer, .txt Vaibhav Gupta CEO, BoundaryML
- 3. 3 | ©Copyright 2024 Zilliz 3 Star Milvus for a chance to win a prize tonight!
- 4. 4 | ©Copyright Zilliz 4 01 Introduction to Milvus and Vector Databases
- 5. 5 | ©Copyright Zilliz 5
- 6. 6 | ©Copyright Zilliz 6 Why is Semantic Search Difficult?
- 7. 7 | ©Copyright Zilliz 7 or Apple or Rising dough or Change car tire Rising Dough Proofing Bread ✔ ❌ Semantic Search vs. Lexical Search
- 8. 8 | ©Copyright Zilliz 8 Why is Semantic Search Important? 10% Other newly generated data in 2025 will be unstructured data 90% Data Source: The Digitization of the World by IDC
- 9. 9 | ©Copyright Zilliz 9 Solution: Deep Learning Similarity Search
- 10. 10 | ©Copyright Zilliz 10 Embeddings and Vector Spaces
- 11. 11 | ©Copyright Zilliz 11 New Challenge: Search in Vector Spaces How to Index and Search? ● High-dimensional ● > 1000 dims How to Scale? ● 10-100 million vectors? ● Billions? ● Trillions? ● Billions of users? Multiple Data Types? ● Text ● Images ● Audio ● Graphs ● …
- 12. 12 | ©Copyright Zilliz 12 About Milvus Milvus is an open-source vector database for GenAI projects. pip install on your laptop, plug into popular AI dev tools, and push to production with a single line of code. 30K GitHub Stars 66M Docker Pulls 400 Contributors 2.7K Forks Easy Setup Pip-install to start coding in a notebook within seconds Integration Plug into OpenAI, Langchain, LlmaIndex, and many more Reusable Code Write once, and deploy with one line of code into the production environment Feature-rich Dense & sparse embeddings, filtering, reranking and beyond
- 13. 13 | ©Copyright 2024 Zilliz 13 Milvus Users
- 14. 14 | ©Copyright 2024 Zilliz 14 14 | © Copyright 10/22/23 Zilliz 14 | © Copyright 2024 Zilliz github.com/milvus-io/milvus zilliz.com/learn/generative-ai
- 15. 15 | ©Copyright Zilliz 15 02 Constrained Sampling from LLMs
- 16. 16 | ©Copyright Zilliz 16 Basic Multinomial Sampling from LLMs Willard and Louf, 2023
- 17. 17 | ©Copyright Zilliz 17 chessboard (environment) context move (action) image instruction JSON instruction code Agents Information Extraction Coding Assistant Classification / Regression question domain class or number
- 18. 18 | ©Copyright Zilliz 18 Structured Output by Prompting “Extract the main details from the following resume and format your answer as JSON. Please make it well-formatted - Iʼd be ever so grateful. Thanks, ta!ˮ Key Idea: Instruct the model to output a given format.
- 19. 19 | ©Copyright Zilliz 19 Structured Output by Guided Sampling Key Idea: Mask logits of tokens that are not valid next tokens Willard and Louf, 2023
- 20. 20 | ©Copyright Zilliz 20 One Implementation: Finite State Machine RegEx r’^d*(.d+)?$’ Key Idea: ● Build FSM of valid strings ● Index valid next tokens for each state in machine. ● Run FSM while generating output, using valid next tokens to produce mask on logits. Mask(state)
- 21. 21 | ©Copyright Zilliz 21 Simple Example Sampled: “ˮ State: 0 Sampled: “.2ˮ State: 3 Sampled: “1ˮ State: 1 RegEx / Grammar: “All numbers with only digits and decimal point.ˮ Tokens (vocab.): “Aˮ, “.ˮ, “42ˮ, “.2ˮ, “1ˮ
- 22. 22 | ©Copyright Zilliz 22 Outlines Library (from .txt)
- 23. 23 | ©Copyright Zilliz 23 Outlines Library (from .txt)
- 24. 24 | ©Copyright Zilliz 24 Try it Out Yourself! https://github.com/boundaryml/baml https://github.com/dottxt-ai/outlines
- 25. 25 | ©Copyright Zilliz 25 T H A N K Y O U
- 26. 26 | ©Copyright 2024 Zilliz 26 https://milvus.io/discord https://github.com/milvus-io/milvus https://x.com/milvusio https://www.linkedin.com/company/the-milvus-project LET’S STAY CONNECTED! Stefan Webb Developer Advocate, Zilliz
- 27. 27 | ©Copyright 2024 Zilliz 27 Become a Speaker! Interesting in speaking at and/or sponsoring a Zilliz Unstructured Data Meetup? Fill out this form 🎤🎤🎤
- 28. 28 | ©Copyright 2024 Zilliz 28 Join us at our next meetup! lu.ma/unstructured-data-meetup Nov 13, South Bay, Location TBD “Evaluating Retrieval-Augmented Generationˮ Nov 19, San Francisco, GitHub Office “Structured Output from Unstructured Input: Part 2ˮ