Enabling GenAI Breakthroughs with Knowledge Graphs

© 2024 Neo4j, Inc. All rights reserved.
Generative AI
workshop
Erik Bijl, Tom Geudens-Breda 2024/03/13
1

Who is he?
Tom Geudens (he/him/his)
(Tom) - [:LIVES_NEAR] ->(`Brussels, Belgium`)
(Tom) - [:USED_TO_BE] -> (ÌDMS DBA`)
(Tom) - [:IS_AUTHOR_OF] -> (Àn O'Reilly Book`)
(Tom) - [:IS_PART_OF] -> (ÈMEA Field Team`)
(Tom) - [:LOVES] -> (Neo4j)
2

Who is he?
Erik Bijl (he/him/his)
(Erik) - [:LIVES_IN] -> (`Zwolle, Netherlands`)
(Erik) - [:USED_TO_BE] -> (`Data Scientist`)
(Erik) - [:WORKED_IN] -> (`Finance`)
(Erik) - [:IS_PART_OF] -> (`EMEA Field Team`)
(Erik) - [:LOVES] -> (Neo4j)
3

Code of Conduct
● Speak up or be quiet forever. Seriously, this is supposed to
be an interactive session (you could have stayed home and
watched the video afterwards otherwise ;-).
● If you are stuck, ask for help. Don't worry too much about it
though, you will get all the materials and can go through
them at your own pace afterwards.
● Bear with us when (not if ;-) technology fails.
● Have fun!
4

Why are you in
THIS workshop?
5

Quick poll (by show of hands)
6

AI is critical to the success of my
company over the next 5 years
7

Results
94% of business leaders agreed with that statement in
october 2022 (Deloitte study)!
8
That actually wasn't the thing about that report that I found
interesting.

This is
9

Retrieval-augmented generation
RAG augments LLMs by retrieving up-to-date, contextual
external data to inform responses:
● Reduce hallucinations with veriﬁed data
● Provide domain-speciﬁc, relevant responses
● Enable traceability back to sources
10

RAG Diagram
11
Opportunity
Your Application
User
1
Q: What…?
5
The answer is…
Embedding
Generator
“Q”
“E”
2
Knowledge Graph
With Vector Support
Vector
search
for “E”
Relevant
Documents and
Context
3
LLM
Documents and
Context
Summary
4
+ “Q”

RAG with
12
Find similar documents
and content
Identify entities
associated to content and
patterns
in connected data
Improve GenAI inferences
and insights. Discover new
relationships and entities
Unify vector search, knowledge graph and data science
capabilities to improve RAG quality and effectiveness
Vector Search
Graph Data
Science
Knowledge
Graph

© 2024 Neo4j, Inc. All rights reserved. 13
Today

Today
we are choosing
clothes
14

For cousin Nina
Nina likes t-shirts
15

Traditional approach
● Line up options
● Let Nina pick
● Add some
accessories
16
That works, but what if
Nina isn't there, what if
you want to buy Nina a
present?

The mission - a graph based AI
shopping/fashion assistant
because Nina is worth it
17
Steps
● Knowledge graph building
● Semantic search
● Personalised search
● Recommendation engine
● Fashion assistant

Tools
18

Knowledge
graph building
19

Dataset
Kaggle H&M personalized fashion recommendations
https://www.kaggle.com/competitions/h-and-m-personalized-
fashion-recommendations/data
Changes
● Normalized into a graph model (coming right up)
● Added random (!) names
20

Model (labels and types)
21

Model (properties)
22

Model (rant)
23

Hammer Time
● Go to the Jupyter Lab environment and open the
genai_workshop.ipynb ﬁle.
We'll run it together, step by step
● For those not familiar with notebooks, when you run a step,
please wait for the [*] to turn into a [<number>], most steps
will also have output to look at.
● If you want to run ahead, please wait at the point where we
will switch to the Neo4j Browser (that's just before Vector
Search).
24

Exploring the graph
In the notebook, click the link to open the Neo4j Browser at
https://browser.neo4j.io
Connect url: <database url>
Username: attendeeXX
Password: attendeeXX
25

Executing a browser guide
● Cut and paste the browser guide link (:play included) from
the notebook.
● Execute with the blue arrow.
26

the notebook.
27

One is one and all alone
Find a single person by id
28
MATCH (c:Customer WHERE c.id =
"daae10780ecd14990ea190a1e9917da33fe96cd8cfa5e80b67b4600171aa77e0")
RETURN c;
001
Find a single person by name
MATCH (c:Customer WHERE c.name = "Nina Massey)
RETURN c;
002

And it's a customer
29
Find a single customer by name
MATCH sg=(c:Customer)-[:PURCHASED]->(:Article)
WHERE c.name = "Nina Massey"
RETURN sg;
003

And that's not without value
It may so far not look interesting, but just with this data we
could determine
● How loyal a customer Nina is
● How big a spender Nina is
● What Nina's favourite colours are
● …
Let's try the colours.
30

Showing your true colours
31
Find Nina's favourite colours
MATCH (c:Customer)-[:PURCHASED]->(a:Article)
RETURN a.colour AS colour, count(*) AS occurences
ORDER BY occurences DESC;
004

From article to product
32
Buying products
MATCH
sg=(c:Customer)-[:PURCHASED]->(:Article)-[:VARIANT_OF]->(:Product)
RETURN sg;
005

More value
It may still not look very exciting but now we can determine
● Which products Nina has bought multiple times
→ and maybe if that coincided with a promotion
● In which order products are bought
→ and maybe if there is a pattern there that we can also
ﬁnd with other customers
● …
33

Compulsive buying
34
Find Nina's favourite products
MATCH
(c:Customer)-[:PURCHASED]->(:Article)-[:VARIANT_OF]->(p:Product)
RETURN p.name AS productname, count(*) AS occurences
ORDER BY occurences DESC;
006
do make a mental note of the kind of sweater Nina likes

Location, location, location
35
Where to ﬁnd Nina
MATCH
sg=(c:Customer)-[:PURCHASED]->(:Article)-[:LOCATED_IN]->(:Department)
RETURN sg;
007

More value still?
The department information could be interesting but less so.
It's way more likely that Nina buys in a department because she
likes the products there than the other way around.
Having said that, when considering many customers it could
be an outlier detector.
For the rest of the session we'll ignore department.
36

Semantic
search
37

Vector
● an arrow
● direction and magnitude (strength)
38

Vector - array of numbers
● is a way to quantify the direction and magnitude in numbers
● for example in a 2D space, x may indicate the horizontal
direction, y the vertical direction and the magnitude is
calculated by the formula
39

Vector - embedding
● represent complex data in a dense numerical form
● embed the data in a high-dimensional space where similar
data is close together
Whilst these vectors can be huge (OpenAI now has a 3072
dimension embedding model), this is nothing compared to the
complexity of what they embed. An embedding is a-massive-
dimension reduction.
40

Vector - similarity
Euclidean-distance based
41
vector point
query
nearest 4
Cosine-direction based

Vector - index
Neo4j implements the Hierarchical Navigable Small World
algorithm to do efﬁcient k-ANN (aproximate nearest
neighbours) searches.
42
Backless
blouse
Tie-back
shirt
Sleeve-
less crop
CREATE VECTOR INDEX …

Vector - search
43
“Halter neck top”
CALL db.index.vector.queryNodes …

Just so it's clear
A full text index search can not ﬁnd King unless King is in the
text. Context is largely ignored.
A vector index search will ﬁnd Queen, Prince, Baron, Warlord,
<and so on> to be to some degree similar to what you asked
but might also weave in some unexpected results. Context is-
well-King.
44

Hammer Time
● Go back to the Jupyter Lab environment.
We'll continue to run it together, step by step
● If you want to run ahead, please wait at the point where we
will switch to the Neo4j Browser again (that's just after the
Semantic Search with Context heading).
45

Personalised
search
46

the notebook.
47

the notebook.
48

HisHer story
Purchase history of a single customer
49
MATCH
(c:Customer)-[pc:PURCHASED]->(:Article)-[:VARIANT_OF]->(p:Product)
RETURN c.name AS name,
pc.transactiondate AS transactiondate,
p.name AS product,
p.description AS description
ORDER BY transactiondate DESC;
001

Nina won the lottery
Shopping spree
50
MATCH
collect(p.name) AS spree
ORDER BY size(spree) DESC;
002

Double trouble
Eliminate doubles
51
MATCH
collect(DISTINCT p.name) AS spree
ORDER BY size(spree) DESC;
003

WITH or without you
Pipeline building
52
MATCH
WITH c.name AS name,
ORDER BY size(spree) DESC LIMIT 1
RETURN transactiondate AS dateofinterest;
004

Similar customers also buy X (part one)
53
// determine the date of interest
MATCH
WITH c.name AS name,
pc.transactiondate AS dateofinterest,
ORDER BY size(spree) DESC LIMIT 1;
...
005

Similar customers also buy X (part two)
54
...
// what other products do customers buy
MATCH
(c)-[pc:PURCHASED]->(:Article)<-[:PURCHASED]-(:Customer)-[:PURCHASED]
->(:Article)-[:VARIANT_OF]->(p:Product)
WHERE pc.transactiondate = dateofinterest
RETURN p.name AS product,
count(*) AS commonPurchaseScore,
p.description AS description
ORDER BY commonPurchaseScore DESC;
005

I know, I know
that a lot of time today is spend on working with the
knowledge graph, with the data
55
But then there are also lot of questions along the lines of
● Why should I integrate a knowledge graph with the LLM,
have you seen the results of what <latest model> can do?
→ yes indeed and they are often crap if you dig just below the
surface

I know, I know
that a lot of time today is spend on working with the
knowledge graph, with the data
56
Or rants along the lines of
● You should provide proof and benchmarks that integrating
a knowledge graph with the LLM provides better results!
→ yes indeed and we do but what's the point if you do not
understand the value of a knowledge graph to start with

Hammer Time
● If you want to run ahead, please wait at Augmenting
Semantic Search with Knowledge Graph Inference & ML.
57

Recommendation
engine
58

Graph Transactional
So far we always queried the graph database in the following
fashion
● Find starting point (Nina)
● Walk the graph from the starting point
● Stay local to the starting point
→ This is what is called graph transactional querying, used in
hundreds of real time use cases.
59

Graph Data Science
You can also consider the graph as a whole and there are many
algorithms that do that
● (Lary) Pagerank
● (Edsger) Dijkstra's pathﬁnding
● (Vincent) Blondel's Louvain community detection
● <many others>
→ This is what is called graph data science, used in hundreds
of analytical use cases and to enhance the graph for the real
time use cases.
60

Our overview
61

Crazy fan art overview
62
It's a web page,
each block links to
the documentation
for that algorithm.
Totally insane,
but 💖

Remember this?
63

It actually looks like this now
64

And we want it to look like this
65

Hammer Time
● If you want to run ahead, please wait at LLM For Generating
Grounded Content.
66

Just so it's clear
There are several possible approaches to create the
CUSTOMERS_ALSO_LIKE relationship. The embedding is
deﬁnitely not the easiest ;-), but
● ﬁts the context (earlier on we saw text embeddings, now we
used node embeddings)
● allows the inclusion of properties (even though we didn't do
that)
Important to understand is that the similarity here is based on
what was bought together, not on what the article is!
67

Fashion
Assistant
68

Pulling it all together
So far you have done
a vector search
and enhanced that
with a personalized search
and you also know what you could recommend.
69
Allow me to introduce you to our AI Fashion Assistant Sam,
who will combine all your hard work. Sam is pretty new to this
though and needs a bit of prompting.

Prompting Sam
You are a personal assistant named Sam for a fashion, home, and beauty company called
HRM.
write an email to {customerName}, one of your customers, to promote and summarize
products relevant for them given the current season / time of year:{timeOfYear}.
Please only mention the products listed below. Do not come up with or add any new
products to the list.
Each product comes with an https `url` field. Make sure to provide that https url with
descriptive name text in markdown for each product.
---
# Relevant Products:
{searchProds}
# Customer May Also Be Interested In the following
(pick items from here that pair with the above products well for the current season /
time of year: {timeOfYear}.
prioritize those higher in the list if possible):
{recProds}
---
70

Sam's insides
71

Sam's insides
72
Personalized Search
Recommendations
Personal information
search prompt
customer id
time of year

Hammer Time
We'll continue to run it together to the end, step by step
73

Resources
Original session:
https://github.com/neo4j-product-examples/genai-workshop
Today's session:
https://github.com/tomgeudens/jupyter-notebooks/tree/main/genaihm
Today's data:
https://github.com/tomgeudens/trainingdata/tree/main/genaihm
74

Thank You!
erik.bijl@neo4j.com
tom.geudens@neo4j.com
75

Enabling GenAI Breakthroughs with Knowledge Graphs

Recommended

Recommended

More Related Content

Similar to Enabling GenAI Breakthroughs with Knowledge Graphs

Similar to Enabling GenAI Breakthroughs with Knowledge Graphs (20)

More from Neo4j

More from Neo4j (20)

Recently uploaded

Recently uploaded (20)

Enabling GenAI Breakthroughs with Knowledge Graphs