Download as PDF, PPTX
![51 | © Copyright Zilliz
51
Vector Similarity Measures: L2 Euclidean)
Queen = [0.3, 0.9]
King = [0.5, 0.7]
d(Queen, King) = √(0.3-0.5)2
+ (0.9-0.7)2
= √(0.2)2
+ (0.2)2
= √0.04 + 0.04
= √0.08 ≅ 0.28](https://image.slidesharecdn.com/09-03-2024unstructureddataandaidiscussion-240903211441-86381ae5/85/09-03-2024_UnstructuredDataAndAIDiscussion-pdf-51-320.jpg)
![52 | © Copyright Zilliz
52
Vector Similarity Measures: Inner Product IP
Queen = [0.3, 0.9]
King = [0.5, 0.7]
Queen · King = (0.3*0.5) + (0.9*0.7)
= 0.15 + 0.63 = 0.78](https://image.slidesharecdn.com/09-03-2024unstructureddataandaidiscussion-240903211441-86381ae5/85/09-03-2024_UnstructuredDataAndAIDiscussion-pdf-52-320.jpg)
![53 | © Copyright Zilliz
53
Queen = [0.3, 0.9]
King = [0.5, 0.7]
Vector Similarity Measures: Cosine
𝚹
cos(Queen, King) = (0.3*0.5)+(0.9*0.7)
√0.32
+0.92
* √0.52
+0.72
= 0.15+0.63 _
√0.9 * √0.74
= 0.78 _
√0.666
≅ 0.03](https://image.slidesharecdn.com/09-03-2024unstructureddataandaidiscussion-240903211441-86381ae5/85/09-03-2024_UnstructuredDataAndAIDiscussion-pdf-53-320.jpg)
Uploaded byTimothy Spann
09-03-2024_UnstructuredDataAndAIDiscussion.pdf
The document presents an overview of unstructured data and the challenges it poses, highlighting that 90% remains unanalyzed. It discusses the role of vector databases, specifically Milvus, in storing and querying vector embeddings derived from this data, offering solutions for scalability and performance. Milvus is tailored for high-performance search capabilities, emphasizing features like multi-tenancy, advanced filtering, and hybrid search for efficient data management.
More Related Content
Similar to 09-03-2024_UnstructuredDataAndAIDiscussion.pdf
![제 23회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [MBOAX] : ABSA를 활용한 소비자 반응 분석 기반 운영 효율화 대시보드 설계](https://cdn.slidesharecdn.com/ss_thumbnails/3-1boaz23rdconferencemboax-260203102709-9d519923-thumbnail.jpg?width=640&height=640&fit=bounds)
09-03-2024_UnstructuredDataAndAIDiscussion.pdf
- 1. 1 | ©Copyright 2024 Zilliz 1 Unstructured Data and AI Discussion Tim Spann @ Zilliz
- 2.
- 3. 3 | ©Copyright Zilliz 3 01 Introduction
- 4. 4 | ©Copyright 2024 Zilliz 4 4 | © Copyright 10/22/23 Zilliz 4 | © Copyright 2024 Zilliz Tim Spann Principal Developer Advocate, Zilliz tim.spann@zilliz.com https://www.linkedin.com/in/timothyspann/ https://x.com/PaaSDev
- 5. 5 | ©Copyright Zilliz 5 Show Me A Demo
- 6. Unstructured Data isEverywhere Unstructured data is any data that does not conform to a predefined data model. Currently, 90% of unstructured data is never analyzed. Images Videos and more! Text
- 7. 7 | ©Copyright Zilliz 7 …and cannot process increasingly growing unstructured data Data Source: The Digitization of the World by IDC 20% Other newly generated data in 2025 will be unstructured data 80%
- 8. The challenge ofunstructured data ● Problem: Unstructured data comes in lots of forms, no easy way to interact with it all ● Solution: Vector embeddings ● How: Neural networks e.g. embedding models Vector Databases
- 9. 9 | ©Copyright Zilliz 9 02 Overview of Vector Databases
- 10. Why a VectorDatabase? •Vector database • Advanced filtering (filtered vector search, chained filters) • Hybrid search (e.g. full text + dense vector) • Durability (any write in a db is durable, a library typically only supports snapshotting) • Replication / High Availability • Sharding • Aggregations or faceted search • Backups • Lifecycle management (CRUD, Batch delete, dropping whole indexes, reindexing) • Multi-tenancy • Vector search library • High-performance vector search •How do I support different applications? • High query load • High insertion/deletion • Full precision/recall • Accelerator support (GPU, FPGA) • Billion-scale storage Purpose-built to store, index and query vector embeddings from unstructured data.
- 11. Vn, 1 … … … 1 2 3 4 5 Transform into Vectors UnstructuredData Images User Generated Content Video Documents Audio Vector Embeddings Perform Approximate Nearest Neighbor Similarity Search Perform Query Get Results Store in Vector Database How Similarity Search Works
- 12. 1 2 2024 A vector databasestores embedding vectors and allows for semantic retrieval of various types of unstructured data. Vector Database: Making Sense of Unstructured Data
- 13. 13 | ©Copyright Zilliz 13 Do you really need a Vector Database? • 50M100M vectors • PostgreSQL, ElasticSearch, Big Query, MongoDB, etc with ANNS plug-ins Existing Solutions Vector Databases • Purpose-built for vectors top support the requirements and lifecycle of vectors • Billion+ scale • CRUD, real-time search, top-k/range/hybrid search, multi-modal, mulit-vector query, distributed • Semantic Search is core to your business ANN Libraries • FAISS, ANNOY, HNSW • Supports 1M vectors • Good for prototyping Vector Databases are purpose-built to handle indexing, storing, and querying vector data.
- 14. 14 | ©Copyright Zilliz 14 03 A Quick Introduction to Milvus
- 15. 15 | ©Copyright Zilliz 15 Milvus Features Multi-Tenancy Hardware- Accelerated Compute Support Python, Java, Golang, NodeJS Milvus Lite, K8, Zilliz Cloud, Docker Scalable and Elastic Architecture Diverse Index Support Versatile Search Capabilities Tunable Consistency
- 16. 16 | ©Copyright Zilliz 16 Technologies for various types of Use cases Compute Types Designed for various compute powers, such as AVX512, Neon for SIMD, quantization cache-aware optimization and GPU Leverage strengths of each hardware type, ensuring high-speed processing and cost-effective scalability for different application needs Search Types Support multiple types such as top-K ANN, Range ANN, sparse & dense, multi-vector, grouping, and metadata filtering Enable query flexibility and accuracy, allowing developers to tailor their information retrieval needs Multi-tenancy Enable multi-tenancy through collection and partition management Allow for efficient resource utilization and customizable data segregation, ensuring secure and isolated data handling for each tenant Index Types Offer a wide range of 15 indexes support, including popular ones like Hierarchical Navigable Small Worlds HNSW, PQ, Binary, Sparse, DiskANN and GPU index Empower developers with tailored search optimizations, catering to performance, accuracy and cost needs
- 17. 17 | ©Copyright Zilliz 17 What is Milvus/Zilliz ideal for? • Advanced filtering • Hybrid search • Multi-vector Search • Durability and backups • Replications/High Availability • Sharding • Aggregations • Lifecycle management • Multi-tenancy • High query load • High insertion/deletion • Full precision/recall • Accelerator support GPU, FPGA • Billion-scale storage Purpose-built to store, index and query vector embeddings from unstructured data at scale.
- 18. 18 | ©Copyright Zilliz 18 Milvus: From Dev to Prod AI Powered Search made easy Milvus is an Open-Source Vector Database to store, index, manage, and use the massive number of embedding vectors generated by deep neural networks and LLMs. contributors 285 stars 29K downloads 50M forks 2.8K
- 19. 19 2024 Higher Scalability 10B vectors of1536 dimensions in a single Milvus/Zilliz Cloud instance 100B vectors in one of the largest deployment
- 20. Milvus: decoupling computationand storage
- 21. 21 2024 Indexes Most of thevector index types supported by Milvus use approximate nearest neighbors search ANNS, ● HNSW: HNSW is a graph-based index and is best suited for scenarios that have a high demand for search efficiency. There is also a GPU version GPU_CAGRA, thanks to Nvidiaʼs contribution. ● FLAT: FLAT is best suited for scenarios that seek perfectly accurate and exact search results on a small, million-scale dataset. There is also a GPU version GPU_BRUTE_FORCE. ● IVF_FLAT: IVF_FLAT is a quantization-based index and is best suited for scenarios that seek an ideal balance between accuracy and query speed. There is also a GPU version GPU_IVF_FLAT. ● IVF_SQ8: IVF_SQ8 is a quantization-based index and is best suited for scenarios that seek a significant reduction on disk, CPU, and GPU memory consumption as these resources are very limited. ● IVF_PQ: IVF_PQ is a quantization-based index and is best suited for scenarios that seek high query speed even at the cost of accuracy. There is also a GPU version GPU_IVF_PQ.
- 22. 22 2024 Indexes Continued. ● SCANN:SCANN is similar to IVF_PQ in terms of vector clustering and product quantization. What makes them different lies in the implementation details of product quantization and the use of SIMD Single-Instruction / Multi-data) for efficient calculation. ● DiskANN: Based on Vamana graphs, DiskANN powers efficient searches within large datasets.
- 23. 23 | ©Copyright Zilliz 23 04 Consume and Ingest Data
- 24. 24 | ©Copyright Zilliz 24 DATA!!!!
- 25. 25 | ©Copyright Zilliz 25 DATA
- 26. 26 | ©Copyright Zilliz 26 05 Building a local RAG application
- 27. 27 | ©Copyright Zilliz 27 Vector embeddings are something computers can understand
- 28. 2 8 Retrieval-Augmented Generation (RAG) 2024 Atechnique that combines the strength of retrieval-based and generative models: ● Improve accuracy and relevance ● Eliminate hallucination ● Provide domain-specific knowledge
- 29. 2 9 RAG : aneconomic perspective 2024 A business model that bridges public data and private data ● Data sovereignty ● You can't and shouldn't give your private data to others
- 30. 30 | ©Copyright Zilliz 30
- 31. 31 | ©Copyright Zilliz 31 Embeddings Models
- 32. 32 | ©Copyright Zilliz 32 06 Q & A
- 34. 34 | ©Copyright Zilliz 34 | © Copyright Zilliz 34 RESOURCES
- 35. 35 | ©Copyright Zilliz 35 Vector Database Resources Give Milvus a Star! Chat with me on Discord! https://github.com/milvus-io/milvus
- 36. 36 Unstructured Data Meetup https://www.meetup.com/unstructured-data-meetup-new-york/ Thismeetup is for people working in unstructured data. Speakers will come present about related topics such as vector databases, LLMs, and managing data at scale. The intended audience of this group includes roles like machine learning engineers, data scientists, data engineers, software engineers, and PMs. This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.
- 37. 37 | ©Copyright Zilliz 37 https://zilliz.com/learn/generative-ai
- 39.
- 40.
- 41.
- 42.
- 43. 43 | ©Copyright 2024 Zilliz 43 43 This week in Milvus, Towhee, Attu, GPT Cache, Gen AI, LLM, Apache NiFi, Apache Flink, Apache Kafka, ML, AI, Apache Spark, Apache Iceberg, Python, Java, Vector DB and Open Source friends. https://bit.ly/32dAJft https://github.com/milvus-io/milvus AIM Weekly by Tim Spann
- 44. 44 | ©Copyright 2024 Zilliz 44 milvus.io github.com/milvus-io/ @milvusio @paasDev /in/timothyspann Connect with me! Thank you!
- 45. 45 | ©Copyright 2024 Zilliz 45
- 46. 46 | ©Copyright 2024 Zilliz 46
- 47. 47 | ©Copyright 2024 Zilliz 47 Join us at our next meetup! meetup.com/unstructured-data-meetup- new-york/
- 48. 48 | ©Copyright Zilliz 48 T H A N K Y O U
- 49. 49 | ©Copyright Zilliz 49 05 What is Similarity Search?
- 50. 50 | ©Copyright Zilliz 50 Image from Nvidia Vector Search Overview
- 51. 51 | ©Copyright Zilliz 51 Vector Similarity Measures: L2 Euclidean) Queen = [0.3, 0.9] King = [0.5, 0.7] d(Queen, King) = √(0.3-0.5)2 + (0.9-0.7)2 = √(0.2)2 + (0.2)2 = √0.04 + 0.04 = √0.08 ≅ 0.28
- 52. 52 | ©Copyright Zilliz 52 Vector Similarity Measures: Inner Product IP Queen = [0.3, 0.9] King = [0.5, 0.7] Queen · King = (0.3*0.5) + (0.9*0.7) = 0.15 + 0.63 = 0.78
- 53. 53 | ©Copyright Zilliz 53 Queen = [0.3, 0.9] King = [0.5, 0.7] Vector Similarity Measures: Cosine 𝚹 cos(Queen, King) = (0.3*0.5)+(0.9*0.7) √0.32 +0.92 * √0.52 +0.72 = 0.15+0.63 _ √0.9 * √0.74 = 0.78 _ √0.666 ≅ 0.03
- 54. 54 | ©Copyright Zilliz 54 osschat.io
- 55. 55 | ©Copyright Zilliz 55