How Milvus allows you to run Full Text Search

1 | © Copyright 8/16/23 Zilliz
Full Text Search with Milvus
2.5
Unstructured Data Meetup | Jan. 23

Stephen Batifol
Developer Advocate, Zilliz / Milvus
About Me
stephen.batifol@zilliz.com
linkedin.com/in/stephen-batifol/
@stephenbtl

● Pip-install on your laptop
● Plug into your favorite AI dev tools
● Push to production with a single line of code
Easy to start

2024
Milvus Lite Milvus Standalone Milvus Distributed
● Ideal for prototyping,
small scale
experiments.
● Easy to set up and
use, pip instally
pymilvus
● Scale to ≈1M vectors
● Run on K8s
● Load balancer and
Multi-Node
Management
● Scaling of each
component
independently
● Scale to 100B
vectors
● Single-Node
Deployment
● Bundled in a single
Docker Image
● Supports Primary/
Secondary
● Scale up to 100M
vectors
Ready to scale 🚀
Write your code once, and run it everywhere, at scale!
● API and SDK are the same

| © Copyright 8/16/23 Zilliz
5
What is Full Text Search?

● Advanced document retrieval based on specific terms
and phrases
● Complements semantic search by catching precise term
matches
● Automatically converts text to sparse embeddings
● Uses BM25 algorithm for intelligent relevance scoring
● Perfect for RAG applications needing precise term
matching
What is Full Text Search?

● Hybrid Search can improve RAG search quality
● Dense Embeddings are impressive but can be limited:
○ Lack of explainability
○ Suboptimal performance with:
■ Long-tail queries
■ Rare terms
● Pure keyword matching (like BM25) sometimes
outperforms sophisticated models in domain-specific
scenarios
Why does it matter?

● Vector search is great for semantic
understanding
○ Can miss exact keyword matches
● Run two separate systems
○ Vector DB for semantic search
○ Elasticsearch/similar for keyword search
Results in complex architecture and
operational overhead
The Search Dilemma

Future of search is combining different search paradigms
1. Semantic Search
2. Keyword Search
3. Filtering
All, in one unified platform
Our Vision is more than just Vectors

Letʼs take a step back

TFIDF
Full Text Search algorithm based on term and doc
weight:
● Term Frequency measures the weight of
document: the more terms it includes, the more
important
● Inverse Document Frequency measures the
weight of terms: the less common it is in the doc
base, the more important

BM25
Improvement over the TFIDF algo
Term Frequency Saturation:
● Prevent words that occurs too frequently from dominating the score
● Document Length Normalization
● Reduces TFIDFʼs bias towards long documents

BM25
● Simple implementation
● Good Retrieval performance
○ Queries Per Second
○ Accuracy
● Used by others in the industry
○ ElasticSearch
○ Apache Lucene
○ MongoDB Atlas Search
○ …

Why do we care about BM25?
We want to:
● Augment the Search Quality of Embedding based
Semantic Search
● Provide Search with more emphasis on Keyword
Matching
● Easy Hybrid Search of BM25  Dense Embeddings
in a single system.

Sparse Vectors

● Super high dimensional vectors with only a few being non-zero.
● Usually represented as a map of {non_zero_dim: actual_value}
Sparse Vectors

Due to the extra high dimensionality, L2 or COSINE will most likely not work due to the
Curse of Dimensionality.
All mass are spread on the surface of the high dimension sphere, all vectors tends to
have a same distance to each other, making kNN meaningless.
Sparse Vectors Metric

Computing BM25 using
Sparse Vectors

Old Approach
1. Get a static statistics of word distribution and document length
2. Encode the document and insert into the database
3. When search, encode the query and rank by IP score.
Problem:
● Statistics are static and once trained cannot be updated as documents being
added/removed.
● The user must manage the statistics themselves from the client side.
⇒ This can be annoying for the user!

Old Approach
Documents / Query
Text Analyzer
Stats Management
Vector DB
Build / Search on Sparse
Vector Index
Hello Milvus
“helloˮ, “milvusˮ
Doc
{
“helloˮ: f(TF),
“milvusˮ: f(TF),
}
Query
{
“helloˮ: IDF,
“milvusˮ: IDF,
}

Old Approach
Documents / Query
Text Analyzer
Stats Management
Vector DB
Vector Index
Clientʼs Side

Milvusʼ Approach

Milvusʼ Approach
The user can almost forget about Vectors
● Insert the raw text into Milvus and search using the text.
Milvus takes care of
● Text Analyzing and Tokenization
● Term distribution statistics management
● Document/Query vector encoding
● BM25 based scoring

Milvusʼ Approach
Documents / Query
Text Analyzer
Stats Management
Vector DB
Vector Index
Milvus takes care of everything ✨
Users only need to interact with
their raw data

Converts raw text into a structured, searchable format. An analyzer in Milvus
consists of exactly one tokenizer and zero or more filters.
● Tokenizer: The tokenizer breaks input text into discrete units called tokens.
These tokens could be words or phrases, depending on the tokenizer type.
● Filters: Filters can be applied to tokens to further refine them, for example, by
making them lowercase or removing common words.
Powered by Tantivy.
Text Analyzer

Text Analyzer

Built in Analyzer which supports:
● standard: Suitable for general-purpose text processing, applying standard
tokenization and lowercase filtering.
● english: Optimized for English-language text, with support for English stop
words.
● chinese: Specialized for processing Chinese text, including tokenization
adapted for Chinese language structures.
We also support Custom ones → https://milvus.io/docs/analyzer-overview.md
Text Analyzer

Full Text Search

The Solution
● Unified system for both vector and keyword search
● Built-in sparse-BM25 algorithm
● No need to manually generate embeddings for
keyword search

The Benefits
● Simple operations
○ Single system to maintain
○ Automatic text analysis and tokenization
○ Simplified data consistency
● Higher search accuracy
○ Better search accuracy through a combined approach
○ Better handling of technical terms
● Better system performance
○ Lower latency with unified querying

milvus.io
github.com/milvus-io/
@milvusio
@stephenbtl
/in/stephen-batifol
Thank you

How Milvus allows you to run Full Text Search

How Milvus allows you to run Full Text Search

More Related Content

Similar to How Milvus allows you to run Full Text Search

More from Zilliz

Recently uploaded

How Milvus allows you to run Full Text Search