SlideShare a Scribd company logo
Weaviate-VectorSearchEngine
StackConf 2021
ByLauraHam,CommunitySolutionEngineeratSeMITechnologies
Topics
● Introduction to Weaviate: Vector Search Engine
● Demo
● Weaviate's core features
● Use cases
2
Weaviate
TheAI-basedvectorsearchengine
Weaviate-whyavectorsearchengine?
4
"Wine for seafood"
No products found ...
Traditionalsearchengine
"Wine for seafood"
Vectorsearchengine
Covey Run 2005
Chardonnay
{ "data": [{
"Wine": "Covey Run 2005 Chardonnay",
"Description": "... good with fish ..."
}]}
5
How to do this on
your own data?
And how so
extremely fast?
The question is very
abstract.
How did Google find
exactly this data
node?
How can we predict
this relation?
What'stheproblem?
● 80% to 90% of data is unstructured, that’s about 34 trillion gigabytes!
● We keep storing more and more data but businesses can’t get valuable insights from it.
● 95% of businesses have issues getting value from their unstructured data.
6
7
How to achieve search and automatically
make relations within your own data?
In a easy, fast, secure and scalable way?
8
Weaviate is a cloud-native, modular,
real-time vector search engine
built to scale your machine learning models
9
Weaviate is a cloud-native, modular,
real-time vector search engine
built to scale your machine learning models
10
Weaviate is a cloud-native, modular,
real-time vector search engine
built to scale your machine learning models
11
Weaviate is a cloud-native, modular,
real-time vector search engine
built to scale your machine learning models
12
Weaviate is a cloud-native, modular,
real-time vector search engine
built to scale your machine learning models
13
Variety
Chardonnay
Country
US
Wine
Covey Run
Chardonnay
& description
has category
inCountry
from
Country
from
W
inery
Winery
Covey Run
14
Variety
Chardonnay
Country
US
Wine
Covey Run
Chardonnay
& description
has category
inCountry
from
Country
from
W
inery
Winery
Covey Run
15
Variety
Chardonnay
Country
US
Wine
Covey Run
Chardonnay
& description
has category
i
n
C
o
u
n
t
r
y
fromCountry
f
r
o
m
W
i
n
e
r
y
Winery
Covey Run
16
Variety
Chardonnay
Country
US
Wine
Covey Run
Chardonnay
& description
has category
i
n
C
o
u
n
t
r
y
fromCountry
f
r
o
m
W
i
n
e
r
y
Winery
Covey Run
Chardonnay
Wine
Country
Weaviate-Howdoesitwork?
● Data is stored as high dimensional vectors
● Vector positions capture data meaning and
context
● Pre-trained NLP module for automatic
○ Vectorization
○ Classification
○ Nearest neighbor search
⇒ Weaviateunderstandsthe data
17
Weaviate-Howdoesitwork?
1. Weaviate Machine Learning Modules
● E.g. NLP module trained with fasttext
● This language model represents all words
and concepts in the hyperspace.
18
Weaviate-Howdoesitwork?
2. Automatically vectorize and index your
data
● Weaviate understandsyour data
● When you import data: Weaviate looks at
the language in your data object
● E.g. a Chardonnay is closely related to Wine,
White and the food it fits with
19
Weaviate-Howdoesitwork?
3. Search query
● Your search queries in natural language will
also be vectorized and understood by the
machine learning module of Weaviate
● It is places close to the words and data
object that are semantically related to the
query
● E.g. "Winethatfitswithaseafooddish"
20
Weaviate-Howdoesitwork?
4. Results
● The data objects that are closest to the
search query are retrieved from the dataset
21
Your own data
(e.g.Wines)
Your search query
(e.g.Wineforseafood)
Weaviate-Howdoesitwork?
Pretrained ML model
(e.g.NLP)
22
Results
(e.g.Chardonnay)
Demo
Weaviate-Corefeatures&possibilities
24
Search
howtosearchthevectorspace
Unique combination of
scalar ('non-semantic') search and
ANN or vector ('semantic') search
Classification
howtoclassifyinthevectorspace
Automatically making relations
between data and derive insights
Weaviate Modules
out-of-the-box modules you can use
e.g. Text or Image vectorization,
Question Answering module,
Custom modules or No modules, etc
WeaviateVectorizationModules
● Textvectorizationmodules
○ Weaviate's NLP module: Contextionary
○ Transformer modules: general purpose from Huggingface
● Imagevectorizationmodules
○ To store and search through images
● Question Answering (Q&A) modules
○ To find answers in textual data
● Custom modules
● No modules
○ Vector storage and -search only
25
Q&AmoduleDemo
Open Source
all code and models are open
Weaviate-ScalableVectorSearchEngine
27
End-to-end
Complete solution for industry,
API driven
Cloud-native
Secure
Scalable and fast
Vectorizing and querying big data
Weaviate Modules
out-of-the-box modules you can use
e.g. Text or Image vectorization,
Question Answering module,
Custom modules or No modules, etc
Search
howtosearchthevectorspace
Unique combination of
scalar ('non-semantic') search and
ANN or vector ('semantic') search
Classification
howtoclassifyinthevectorspace
Automatically making relations
between data and derive insights
Exampleusecases
28
SeMITechnologies
29
Team
Etienne Donna Laura Micha Bob
Stefan Marcin Henrique Amar Alicja
TryoutWeaviateyourself!
● Go towww.semi.technology/developers
30
Thankyou!
www.semi.technology
Laura Ham
laura@semi.technology

More Related Content

What's hot

The Semantic Knowledge Graph
The Semantic Knowledge GraphThe Semantic Knowledge Graph
The Semantic Knowledge Graph
Trey Grainger
 
Building Biomedical Knowledge Graphs for In-Silico Drug Discovery
Building Biomedical Knowledge Graphs for In-Silico Drug DiscoveryBuilding Biomedical Knowledge Graphs for In-Silico Drug Discovery
Building Biomedical Knowledge Graphs for In-Silico Drug Discovery
Vaticle
 
Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...
Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...
Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...
Qi Guo
 
How to Build a Recommendation Engine with Neo4j
How to Build a Recommendation Engine with Neo4jHow to Build a Recommendation Engine with Neo4j
How to Build a Recommendation Engine with Neo4j
Neo4j
 

What's hot (20)

Neo4j Presentation
Neo4j PresentationNeo4j Presentation
Neo4j Presentation
 
RDF 개념 및 구문 소개
RDF 개념 및 구문 소개RDF 개념 및 구문 소개
RDF 개념 및 구문 소개
 
Vector databases and neural search
Vector databases and neural searchVector databases and neural search
Vector databases and neural search
 
Introducing Neo4j
Introducing Neo4jIntroducing Neo4j
Introducing Neo4j
 
Learned Embeddings for Search and Discovery at Instacart
Learned Embeddings for  Search and Discovery at InstacartLearned Embeddings for  Search and Discovery at Instacart
Learned Embeddings for Search and Discovery at Instacart
 
Workshop - Build a Graph Solution
Workshop - Build a Graph SolutionWorkshop - Build a Graph Solution
Workshop - Build a Graph Solution
 
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
 
Intro to Neo4j and Graph Databases
Intro to Neo4j and Graph DatabasesIntro to Neo4j and Graph Databases
Intro to Neo4j and Graph Databases
 
The Semantic Knowledge Graph
The Semantic Knowledge GraphThe Semantic Knowledge Graph
The Semantic Knowledge Graph
 
Get Started with the Most Advanced Edition Yet of Neo4j Graph Data Science
Get Started with the Most Advanced Edition Yet of Neo4j Graph Data ScienceGet Started with the Most Advanced Edition Yet of Neo4j Graph Data Science
Get Started with the Most Advanced Edition Yet of Neo4j Graph Data Science
 
MLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
MLOps Virtual Event | Building Machine Learning Platforms for the Full LifecycleMLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
MLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
 
Building Biomedical Knowledge Graphs for In-Silico Drug Discovery
Building Biomedical Knowledge Graphs for In-Silico Drug DiscoveryBuilding Biomedical Knowledge Graphs for In-Silico Drug Discovery
Building Biomedical Knowledge Graphs for In-Silico Drug Discovery
 
Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...
Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...
Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...
 
How to Build a Recommendation Engine with Neo4j
How to Build a Recommendation Engine with Neo4jHow to Build a Recommendation Engine with Neo4j
How to Build a Recommendation Engine with Neo4j
 
Build a Recommendation Engine using Amazon Machine Learning in Real-time
Build a Recommendation Engine using Amazon Machine Learning in Real-timeBuild a Recommendation Engine using Amazon Machine Learning in Real-time
Build a Recommendation Engine using Amazon Machine Learning in Real-time
 
Knowledge Graph Introduction
Knowledge Graph IntroductionKnowledge Graph Introduction
Knowledge Graph Introduction
 
Deep Learning for Semantic Search in E-commerce​
Deep Learning for Semantic Search in E-commerce​Deep Learning for Semantic Search in E-commerce​
Deep Learning for Semantic Search in E-commerce​
 
Denodo Data Virtualization Platform: Overview (session 1 from Architect to Ar...
Denodo Data Virtualization Platform: Overview (session 1 from Architect to Ar...Denodo Data Virtualization Platform: Overview (session 1 from Architect to Ar...
Denodo Data Virtualization Platform: Overview (session 1 from Architect to Ar...
 
Graph and RDF databases
Graph and RDF databasesGraph and RDF databases
Graph and RDF databases
 
Lecture: Question Answering
Lecture: Question AnsweringLecture: Question Answering
Lecture: Question Answering
 

Similar to stackconf 2021 | Weaviate Vector Search Engine – Introduction

Visual Process, an innovative analytical solution by bridging business and da...
Visual Process, an innovative analytical solution by bridging business and da...Visual Process, an innovative analytical solution by bridging business and da...
Visual Process, an innovative analytical solution by bridging business and da...
Avraham CHOUKROUN
 
ESWC SS 2013 - Monday Tutorial Aidan Hogan: Intro to Semantic Web
ESWC SS 2013 - Monday Tutorial Aidan Hogan: Intro to Semantic WebESWC SS 2013 - Monday Tutorial Aidan Hogan: Intro to Semantic Web
ESWC SS 2013 - Monday Tutorial Aidan Hogan: Intro to Semantic Web
eswcsummerschool
 
Data mining & warehousing
Data mining & warehousingData mining & warehousing
Data mining & warehousing
Samoneh Dashti
 
Creating Semantic Mashups Bridging Web 2 0 And The Semantic Web Presentation 1
Creating Semantic Mashups  Bridging Web 2 0 And The Semantic Web Presentation 1Creating Semantic Mashups  Bridging Web 2 0 And The Semantic Web Presentation 1
Creating Semantic Mashups Bridging Web 2 0 And The Semantic Web Presentation 1
jward5519
 
Creating Semantic Mashups Bridging Web 2 0 And The Semantic Web Presentation 1
Creating Semantic Mashups  Bridging Web 2 0 And The Semantic Web Presentation 1Creating Semantic Mashups  Bridging Web 2 0 And The Semantic Web Presentation 1
Creating Semantic Mashups Bridging Web 2 0 And The Semantic Web Presentation 1
jward5519
 
Making the Most of Customer Data
Making the Most of Customer DataMaking the Most of Customer Data
Making the Most of Customer Data
WSO2
 

Similar to stackconf 2021 | Weaviate Vector Search Engine – Introduction (20)

Visual Process, an innovative analytical solution by bridging business and da...
Visual Process, an innovative analytical solution by bridging business and da...Visual Process, an innovative analytical solution by bridging business and da...
Visual Process, an innovative analytical solution by bridging business and da...
 
Ragic Quick Guide: Frequently Asked Questions & Answers
Ragic Quick Guide: Frequently Asked Questions & AnswersRagic Quick Guide: Frequently Asked Questions & Answers
Ragic Quick Guide: Frequently Asked Questions & Answers
 
Extreme Analytics @ eBay
Extreme Analytics @ eBayExtreme Analytics @ eBay
Extreme Analytics @ eBay
 
Extreme Analytics @ eBay
Extreme Analytics @ eBayExtreme Analytics @ eBay
Extreme Analytics @ eBay
 
Yahoo Microstrategy 2008
Yahoo Microstrategy 2008Yahoo Microstrategy 2008
Yahoo Microstrategy 2008
 
Linked Enterprise Vocabularies
Linked Enterprise VocabulariesLinked Enterprise Vocabularies
Linked Enterprise Vocabularies
 
Data Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and GovernanceData Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and Governance
 
ESWC SS 2013 - Monday Tutorial Aidan Hogan: Intro to Semantic Web
ESWC SS 2013 - Monday Tutorial Aidan Hogan: Intro to Semantic WebESWC SS 2013 - Monday Tutorial Aidan Hogan: Intro to Semantic Web
ESWC SS 2013 - Monday Tutorial Aidan Hogan: Intro to Semantic Web
 
SEO for the Semantic Web
SEO for the Semantic WebSEO for the Semantic Web
SEO for the Semantic Web
 
Data mining & warehousing
Data mining & warehousingData mining & warehousing
Data mining & warehousing
 
Just ask: Designing intent-driven algos
Just ask: Designing intent-driven algosJust ask: Designing intent-driven algos
Just ask: Designing intent-driven algos
 
Enterprise Class WordPress
Enterprise Class WordPressEnterprise Class WordPress
Enterprise Class WordPress
 
The biggest stores on Magento
The biggest stores on MagentoThe biggest stores on Magento
The biggest stores on Magento
 
Google Analytics for Store Owners - Intermediate
Google Analytics for Store Owners - IntermediateGoogle Analytics for Store Owners - Intermediate
Google Analytics for Store Owners - Intermediate
 
Ds01 data science
Ds01   data scienceDs01   data science
Ds01 data science
 
Creating Semantic Mashups Bridging Web 2 0 And The Semantic Web Presentation 1
Creating Semantic Mashups  Bridging Web 2 0 And The Semantic Web Presentation 1Creating Semantic Mashups  Bridging Web 2 0 And The Semantic Web Presentation 1
Creating Semantic Mashups Bridging Web 2 0 And The Semantic Web Presentation 1
 
Creating Semantic Mashups Bridging Web 2 0 And The Semantic Web Presentation 1
Creating Semantic Mashups  Bridging Web 2 0 And The Semantic Web Presentation 1Creating Semantic Mashups  Bridging Web 2 0 And The Semantic Web Presentation 1
Creating Semantic Mashups Bridging Web 2 0 And The Semantic Web Presentation 1
 
Making the Most of Customer Data
Making the Most of Customer DataMaking the Most of Customer Data
Making the Most of Customer Data
 
Running a business on Web Scraped Data
Running a business on Web Scraped DataRunning a business on Web Scraped Data
Running a business on Web Scraped Data
 
Delight + adCenter
Delight + adCenterDelight + adCenter
Delight + adCenter
 

Recently uploaded

Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
mbmh111980
 
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
Alluxio, Inc.
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
Max Lee
 

Recently uploaded (20)

KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
 
INGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by DesignINGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by Design
 
A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data Migration
 
Crafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationCrafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM Integration
 
Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...
Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...
Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
 
Top Mobile App Development Companies 2024
Top Mobile App Development Companies 2024Top Mobile App Development Companies 2024
Top Mobile App Development Companies 2024
 
10 Essential Software Testing Tools You Need to Know About.pdf
10 Essential Software Testing Tools You Need to Know About.pdf10 Essential Software Testing Tools You Need to Know About.pdf
10 Essential Software Testing Tools You Need to Know About.pdf
 
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
 
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfMicrosoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
 
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
iGaming Platform & Lottery Solutions by Skilrock
iGaming Platform & Lottery Solutions by SkilrockiGaming Platform & Lottery Solutions by Skilrock
iGaming Platform & Lottery Solutions by Skilrock
 
Breaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdfBreaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdf
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
 
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
 
Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024
 
Designing for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesDesigning for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web Services
 

stackconf 2021 | Weaviate Vector Search Engine – Introduction