The document introduces the 'Data Prep Kit', an open-source toolkit designed to assist in preparing unstructured data for language learning models (LLMs) through various tasks like de-duplication, text extraction, and PII detection. It also highlights the capabilities of the Milvus vector database, which allows users to seamlessly integrate embedding models and AI tools, facilitating the efficient handling of large datasets. A key focus is on retrieval-augmented generation (RAG) techniques that enhance data relevance and accuracy.

1. 1 | © Copyright 2024 Zilliz
1
Introduction to Data Prep Kit and
Open Source RAG
Tim Spann @ Zilliz

2. 2 | © Copyright 2024 Zilliz
2
2 | © Copyright 10/22/23 Zilliz
2 | © Copyright 2024 Zilliz
Tim Spann
Principal Developer
Advocate, Milvus
https://medium.com/@tspann
https://www.linkedin.com/in/timothyspann/
https://x.com/PaaSDev

3. The challenge of Unstructured Data
● Problem: Unstructured data comes in lots of forms, no
easy way to interact with it all
● Solution: Vector embeddings
● How: Neural networks e.g. embedding models
Vector
Databases

4. Unstructured Data is Everywhere
Unstructured data is any data that does not
conform to a predefined data model.
Currently, 90% of unstructured data is never
analyzed.
Images Videos and
more!
Text

5. Common Tasks when Preparing Data for
LLMs
• Documents
– De-duplication of docs
– Extracting text from PDFs /
Documents
– Removing excess markup
– Tokenizing / Chunking documents
– Assessing document quality
– PII detection
• Code
– Language detection
– Malware detection
– Code quality

6. Say Hello to "Data Prep Kit"
Open source toolkit
Helps with data prep
Handles documents + code
Many ready to use modules
out of the box
Python
Develop on laptop, scale on
clusters
https://github.com/sujee/data-prep-kit-examples/blob/main/dpk-intro/README.md

7. Data Prep Kit

8. 8
Retrieval-Augmented Generation (RAG)
2024
A technique that combines the strength
of retrieval-based and generative
models:
● Improve accuracy and relevance
● Eliminate hallucination
● Provide domain-specific
knowledge

9. Open Source RAG
Data Prep Kit

10. RAG Flow

11. 11 | © Copyright 2024 Zilliz
11
Milvus Vector
Database
Milvus is an open-source vector database
for GenAI projects. pip install on your
laptop, plug into popular AI dev tools, and
push to production with a single line of
code.
30K+
GitHub Stars
66M+
Docker Pulls
400+
Contributors
2.7K+
Forks
Easy Setup
Pip-install to start coding in a notebook within seconds
Integration
Plug into OpenAI, Langchain, LlmaIndex, and many more
Reusable Code
Write once, and deploy with one line of code into the production
environment
Feature-rich
Dense & sparse embeddings, filtering, reranking and beyond

12. 12 | © Copyright 2024 Zilliz
12
New Challenge: Search in Vector Spaces
How to Index and
Search?
● High-dimensional
● > 1000 dims
How to Scale?
● 10-100 million vectors?
● Billions?
● Trillions?
● Billions of users?
Multiple Data Types?
● Text
● Images
● Audio
● Graphs
● …

13. 13 | © Copyright Zilliz
13

14. 14 | © Copyright Zilliz
14
Easy Open RAG Stack Highlighted
Framework
Hardware
Infrastructure
Embedding Models LLMs
Software Infrastructure
Vector Database

15. 15 | © Copyright 2024 Zilliz
15
15
This week in Vector Databases, Gen AI, LLM,
Apache NiFi, Apache Flink, Apache Kafka, ML,
AI, Apache Spark, Apache Iceberg, Python,
Java, Vector DB and Open Source friends.
https://bit.ly/32dAJft
https://github.com/milvus-io/milvus
AIM Weekly by Tim Spann

16. 16 | © Copyright Zilliz
16
T H A N K Y O U