Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and more!

1 | © Copyright 8/16/23 Zilliz
Milvus 2.5 Full-Text Search,
More Powerful Metadata
Filtering, and more!
Unifying Vector and Keyword Search

Stephen Batifol
Developer Advocate, Zilliz / Milvus
About Me
stephen.batifol@zilliz.com
linkedin.com/in/stephen-batifol/
@stephenbtl

● Pip-install on your laptop
● Plug into your favorite AI dev tools
● Push to production with a single line of code
Easy to start

2024
Milvus Lite Milvus Standalone Milvus Distributed
● Ideal for prototyping,
small scale
experiments.
● Easy to set up and
use, pip instally
pymilvus
● Scale to ≈1M vectors
● Run on K8s
● Load balancer and
Multi-Node
Management
● Scaling of each
component
independently
● Scale to 100B
vectors
● Single-Node
Deployment
● Bundled in a single
Docker Image
● Supports Primary/
Secondary
● Scale up to 100M
vectors
Ready to scale 🚀
Write your code once, and run it everywhere, at scale!
● API and SDK are the same

| © Copyright 8/16/23 Zilliz
5
Milvus 2.5

● New Bitmap Index
● Nullable & Default Values in Scalar Fields
● A new Cluster Management UI Beta
● Full Text Search
● …
Whatʼs new in 2.5?

Bitmap Index
● Efficient indexing technique designed to improve query
performance on low-cardinality scalar fields.
○ Example: A column containing gender information with only
two possible values: male and female.
● It consists of two main components: bitmaps and keys
● As a general guideline, bitmap indexes are most
effective when the cardinality of a field is less than
500.

Bitmap Index

Nullable & Default Values in Scalar Fields
● Allows you to set the nullable attribute and default values for
scalar fields, except the primary field.
● For fields marked as nullable=True, you can skip the field when
inserting data
● This is particularly useful if you migrate from another DB ;)

● Cluster Management UI Beta
○ http://<url>:9091/webui/
○ Visualise Milvusʼs runtime environment information.
● Migrated HNSW from hsnwlib to FAISS
● Clustering Compaction Beta
○ Accelerate searches and reduce costs in large collections
And more…

13
There is even more!

14
Our Vision is more than Vectors

Future of search is combining different search paradigms
1. Semantic Search
2. Keyword Search
3. Filtering
All, in one unified platform
Our Vision is more than just Vectors

● Vector search is great for semantic
understanding
○ Can miss exact keyword matches
● Run two separate systems
○ Vector DB for semantic search
○ Elasticsearch/similar for keyword search
Results in complex architecture and
operational overhead
The Search Dilemma

17
Solution:
Full Text Search in Milvus
2.5

Letʼs take a step back

TFIDF
Full Text Search algorithm based on term and doc
weight:
● Term Frequency measures the weight of
document: the more term it includes, the more
important
● Inverse Document Frequency measures the
weight of terms: the less common it is in the doc
base, the more important

BM25
Improvement over the TFIDF algo
Term Frequency Saturation:
● Prevent words that occurs too frequently from dominating the score
● Document Length Normalization
● Reduces TFIDFʼs bias towards long documents

BM25
● Simple implementation
● Good Retrieval performance
○ Queries Per Second
○ Accuracy
● Used by others in the industry
○ ElasticSearch
○ Apache Lucene
○ MongoDB Atlas Search
○ …

Why do we care about BM25?
We want to:
● Augment the Search Quality of Embedding based
Semantic Search
● Provide Search with more emphasis on Keyword
Matching
● Easy Hybrid Search of BM25  Dense Embeddings
in a single system.

Sparse Vectors

● Super high dimensional vectors with only a few being non-zero.
● Usually represented as a map of {non_zero_dim: actual_value}
Sparse Vectors

Due to the extra high dimensionality, L2 or COSINE will most likely not work due to the
Curse of Dimensionality.
All mass are spread on the surface of the high dimension sphere, all vectors tends to
have a same distance to each other, making kNN meaningless.
Sparse Vectors Metric

Computing BM25 using
Sparse Vectors

Old Approach
1. Get a static statistics of word distribution and document length
2. Encode the document and insert into the database
3. When search, encode the query and rank by IP score.
Problem:
● Statistics are static and once trained cannot be updated as documents being
added/removed.
● The user must manage the statistics themselves from the client side.
⇒ This can be annoying for the user!

Old Approach
Documents / Query
Text Analyzer
Stats Management
Vector DB
Build / Search on Sparse
Vector Index
Hello Milvus
“helloˮ, “milvusˮ
Doc
{
“helloˮ: f(TF),
“milvusˮ: f(TF),
}
Query
{
“helloˮ: IDF,
“milvusˮ: IDF,
}

Old Approach
Documents / Query
Text Analyzer
Stats Management
Vector DB
Vector Index
Clientʼs Side

Milvusʼ Approach

Milvusʼ Approach
The user can almost forget about Vectors
● Insert the raw text into Milvus and search using the text.
Milvus takes care of
● Text Analyzing and Tokenization
● Term distribution statistics management
● Document/Query vector encoding
● BM25 based scoring

Milvusʼ Approach
Documents / Query
Text Analyzer
Stats Management
Vector DB
Vector Index
Milvus takes care of everything ✨
Users only need to interact with
their raw data

Converts raw text into a structured, searchable format. An analyzer in Milvus
consists of exactly one tokenizer and zero or more filters.
● Tokenizer: The tokenizer breaks input text into discrete units called tokens.
These tokens could be words or phrases, depending on the tokenizer type.
● Filters: Filters can be applied to tokens to further refine them, for example, by
making them lowercase or removing common words.
Powered by Tantivy.
Text Analyzer

Text Analyzer

Built in Analyzer which supports:
● standard: Suitable for general-purpose text processing, applying standard
tokenization and lowercase filtering.
● english: Optimized for English-language text, with support for English stop
words.
● chinese: Specialized for processing Chinese text, including tokenization
adapted for Chinese language structures.
We also support Custom ones → https://milvus.io/docs/analyzer-overview.md
Text Analyzer

Full Text Search

37
Demo!

The Solution
● Unified system for both vector and keyword search
● Built-in sparse-BM25 algorithm
● No need to manually generate embeddings for
keyword search

The Benefits
● Simple operations
○ Single system to maintain
○ Automatic text analysis and tokenization
○ Simplified data consistency
● Higher search accuracy
○ Better search accuracy through a combined approach
○ Better handling of technical terms
● Better system performance
○ Lower latency with unified querying

milvus.io
github.com/milvus-io/
@milvusio
@stephenbtl
/in/stephen-batifol
Thank you

41
Goal:
The Unstructured Data
Platform

Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and more!

Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and more!

More Related Content

Similar to Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and more!

More from Zilliz

Recently uploaded

Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and more!