Large Language models are amazing but are also black-box models that often fail to capture and accurately represent factual knowledge. Knowledge graphs, by contrast, are structural knowledge models that explicitly represent knowledge and, indeed, allow us to detect implicit relationships. In this talk we will demonstrate how LLMs can be improved by Knowledge Graphs, and how LLM’s can augment Knowledge Graphs. A perfect couple!

1. © 2023 Neo4j, Inc. All rights reserved.
© 2023 Neo4j, Inc. All rights reserved.
LLM 💕 KG: The Perfect Couple
Uniting Large Language Models and Knowledge
Graphs for Enhanced Knowledge Representation
Kristof Neys
Director Graph Data Science Technology

2. © 2023 Neo4j, Inc. All rights reserved.
7/10
20/25
7/10
Top Retail Firms
Top Financial Firms
Top Software Vendors
Anyway You Like It
2
Creator of the Property
Graph and Cypher language
at the core of the GQL ISO
project. Fully integrated Data
Science Library
Thousands of Customers
World-Wide
250,000 developers
worldwide
Industry Leaders use Neo4j
On-Prem
DB-as-a-Service
In the Cloud

3. © 2022 Neo4j, Inc. All rights reserved.
3
Topics
We will cover:
1) Knowledge Graphs
2) Knowledge Graphs + LLMs ~ As a Front end
3) Knowledge Graphs + LLMs ~ As a Back end
4) How to get started on your graph journey

4. © 2023 Neo4j, Inc. All rights reserved.
4
“In a world of infinite content, knowledge
becomes [very] valuable”
Denny Vrandečić, Wikidata

5. © 2023 Neo4j, Inc. All rights reserved.
5
The Core Graph Object:
A Knowledge Graph

6. © 2023 Neo4j, Inc. All rights reserved.
What are Knowledge Graphs?
● Entities can be real-world objects and abstract concepts
● Relationships represent the connections between entities
● Semantic description of the entities and relationships
A knowledge graph is a structured representation of
facts, consisting of entities, relationships and semantic
descriptions

7. © 2023 Neo4j, Inc. All rights reserved.
7
From Data points to a Knowledge Graph
Car
DRIVES
name: “Dan”
born: May 29, 1978
twitter: “@dan”
name: “Ann”
born: Dec 5, 1979
brand: “Volvo”
model: “V90”
description:
since:
Jan 10, 2021
LOVES
LOVES
LIVES_WITH
O
W
N
S
Person Person

8. © 2023 Neo4j, Inc. All rights reserved.
User
:VISITED
Website
User
IPLocation
Website
IPLocation
Website
Website
Website
:VISITED
:VISITED
:VISITED
:USED
:USED
:
U
S
E
D
:
V
I
S
I
T
E
D
:
V
I
S
I
T
E
D
:VISITED
:SAME_AS
Graphs allows you to make implicit
relationships….
….explicit
Graphs….Grow!

9. © 2023 Neo4j, Inc. All rights reserved.
:SAME_AS
User
:VISITED
Website
User
IPLocation
Website
IPLocation
Website
Website
Website
:VISITED
:VISITED
:VISITED
:USED
:USED
:
U
S
E
D
:
V
I
S
I
T
E
D
:
V
I
S
I
T
E
D
:VISITED
User
:SAM
E_AS
:USED
:VISITED
PersonId: 1
PersonId: 1 PersonId: 1
User
PersonId: 2
:VISITED
…and can then group similar nodes…and
create a new graph from the explicit
relationships…
A graph grows organically - gaining
insights and enriching your data
Graphs….Grow!

10. © 2023 Neo4j, Inc. All rights reserved.
Come and see us for the book on Knowledge Graphs

11. © 2023 Neo4j, Inc. All rights reserved.
11
Knowledge Graph + LLM
A marriage made in heaven

12. © 2023 Neo4j, Inc. All rights reserved.
12
Setting the context - From a user point of view

13. © 2023 Neo4j, Inc. All rights reserved.
13
As a front-end…

14. © 2023 Neo4j, Inc. All rights reserved.
14
Architecture
NeoDash
Dashboard UI
Neo4j
Free text input
APOC call Graph representations
Embeddings
Graph visualization of results
NeoDash
Dashboard UI
Persist results
Conceptual flow for a
text-to-graph or graph-to-text
application based on Neo4j +
OpenAI.

15. © 2023 Neo4j, Inc. All rights reserved.
15
Writing Code is soo last year….

16. © 2023 Neo4j, Inc. All rights reserved.
16
As a back-end…

17. © 2023 Neo4j, Inc. All rights reserved.
17
We are not making this up…

18. © 2023 Neo4j, Inc. All rights reserved.
18
Three main ways to let LLM’s work with your Graph

19. © 2023 Neo4j, Inc. All rights reserved.
19
Knowledge Graphs that enhance LLMs

20. © 2023 Neo4j, Inc. All rights reserved.
LLMs Core function…To manipulate strings…
• GPT training focuses primarily on predicting the probability of text sequences and
specifically for predicting which words come next in a sequence
“When you play the
game of thrones….
Input
LLM
…you win or you die.
There is no middle
ground.”
Output

21. © 2023 Neo4j, Inc. All rights reserved.
21
Nothing is perfect…
but don’t push it!

22. © 2023 Neo4j, Inc. All rights reserved.

23. © 2023 Neo4j, Inc. All rights reserved.
There is no need for this…!

24. © 2023 Neo4j, Inc. All rights reserved.
Whereas we all know what The Truth is…

25. © 2023 Neo4j, Inc. All rights reserved.

26. © 2023 Neo4j, Inc. All rights reserved.
It’s all about Knowledge and Accuracy
• Training data can be false
• LLMs: training is focused on self-supervised learning
• Cut-off date: i.e ChatGPT is only trained on pre Sep 2021 data
• Inability to verify sources for answers
• Sensitive to prompt (input) phrasing
• Lack of explainability

27. © 2023 Neo4j, Inc. All rights reserved.
Making stuff up…

28. © 2023 Neo4j, Inc. All rights reserved.
In Summary…

29. © 2023 Neo4j, Inc. All rights reserved.
29
So…, let’s solve this problem!

30. © 2023 Neo4j, Inc. All rights reserved.
How to Help LLMs Do Better?
Few-Shot Learning
Fine-Tuning Grounding
Provide completed
examples “shots” to the AI
as context in prompts.
a.k.a In-Context Learning
Provide additional training
data to better tune GenAI
to your use case
Provide AI with the
information to use for
generating responses

31. © 2023 Neo4j, Inc. All rights reserved.
Grounding
Provide AI with the information to use for generating responses
Best of Both Worlds
Use the language understanding skills of LLMs but control the knowledge/sources it uses
● Improve accuracy & tailor to enterprise and use case
● Reduce hallucinations and other inconsistencies
● Control provenance
Methods
● Internet search, plugins, agents
● Query generation
● Retrieval Augmented Generation (RAG)
Image by GenAI (DALL-E)

32. © 2023 Neo4j, Inc. All rights reserved.
32
Grounding with Retrieval Augmented Generation (RAG)

33. Neo4j, Inc. All rights reserved 2021
33
What’s in it for me…?

34. © 2023 Neo4j, Inc. All rights reserved.
LLMs can help generate more accurate responses by
considering the connections and dependencies within the
graph and mapping new links as new data is identified.
Flexible Schema means it is
easy to grow your knowledge
base whenever new
information is available
Relationships are data
that is used to return
explicit results
Accuracy
Vector Search adds semantic
search

35. © 2023 Neo4j, Inc. All rights reserved.
Knowledge graphs supply the LLM with information about your
company so answers are specific to your business, giving more
context for more accurate responses.
Add a layer of context over
your LLM for accuracy and
specificity
Specificity

36. © 2023 Neo4j, Inc. All rights reserved.
Map relationships between search
results and data source nodes
Represent data sources as nodes
Add metadata or annotations
Verify the enriched responses from your LLM because each piece of
information is linked to its sources and origins.
Explainability

37. © 2023 Neo4j, Inc. All rights reserved.
LLM retrieves and returns information governed by your enterprise security
and access control policies-down to the node level.
Define policies by role or
identity
Integrates with identity and
access management provider
with SSO
Security & Privacy
Build constraints on nodes,
labels, relationships, properties,
specific parts of the graph, and
even traversal depth

38. © 2023 Neo4j, Inc. All rights reserved.
Easily scale with autonomous
clustering
Reliability & Low-latency
Knowledge graphs scale and are battle-tested to thousands of
concurrent users, get answers quickly with incredibly fast query speeds.
Incredibly fast traversals with index
free adjacency

39. © 2023 Neo4j, Inc. All rights reserved.
39
Use LLM to augment Knowledge Graphs
● Named Entity Recognition
● Knowledge Compression

40. © 2023 Neo4j, Inc. All rights reserved.
40
Use LLM to augment Knowledge Graphs:
Named Entity Recognition

41. © 2023 Neo4j, Inc. All rights reserved.
Creating a Knowledge Graph From Unstructured Text is
Difficult
Traditional entity extraction pipelines can be
• resource intensive
• difficult to transfer/generalize:
◦ specialized NLP libraries/tools
◦ domain specific
◦ require subject matter expertise
◦ lots of trial & error
◦ Mix of business rules, machine learning,
and custom hand-rolled logic

42. © 2023 Neo4j, Inc. All rights reserved.
LLMs can Jumpstart Knowledge Graph Creation
LLMs can be prompted to:
1. Extract data from unstructured text
• Entity extraction, including pronouns and
partial references
• Entity disambiguation
• Extraction of semantic relationships
2. Aide or directly perform graph ingestion
• Determine desired output (Pandas DataFrame
etc)
• Structure extracted data (triples, json, csv) to
work with Cypher insert templates
Input Data
LLM
User
Neo4j

43. © 2023 Neo4j, Inc. All rights reserved.
43
Use LLM to augment Knowledge Graphs:
Knowledge Compression

44. © 2023 Neo4j, Inc. All rights reserved.
Text Embeddings are a Key Concept for Knowledge Compression
Consider the question: What is text embedding?
It can be represented / translated as a vector using the OpenAI Embedding API:
[-0.03156438, 0.0013196499, -0.01716885, -0.0008197554, 0.011872382, 0.0036221128,
-0.022915626, -0.005925469, … (1528 more items to go here …]
• This translation process is called Embedding. It is a machine learning process to
convert complex, high-dimensional data e.g. text, image etc. into lower-dimensional
representations(vectors) while preserving essential relationships and structure.
• Embedding is the knowledge for AI as it is produced, understood and used by
various algorithms and AI systems.

45. © 2023 Neo4j, Inc. All rights reserved.
Text Embedding Vectors for Semantic Search
Given a question, find the most relevant documents based on a similarity metric (such as
Cosine Similarity) between vector of the question and vectors of contents.
Moving from keyword search to similarity (semantic) search.
Q: what is text
embedding?
abstractId similarity
456 0.923445
22 0.892114
… ...
Top K by similarity

46. Use LLMs for Graph Enrichment
Neo4j Inc. All rights reserved 2023
46
[0.2322,0.3321,….,0.0021]
[0.3233,0.3543,….,0.0047]
[0.5674,0.2134,….,0.0054]
[0.4565,0.2345,….,0.0342]
[0.8743,0.4343,….,0.0234]
LLM APIs
(OpenAI/ VertexAI/
AWS Bedrock etc.)
Text
Embeddings
Article PDFs
(Unstructured Data)

47. © 2023 Neo4j, Inc. All rights reserved.
47
One last thing…

48. © 2023 Neo4j, Inc. All rights reserved.
48
https://neo4j.com/blog/vector-search-deeper-insights/

49. © 2023 Neo4j, Inc. All rights reserved.

50. © 2023 Neo4j, Inc. All rights reserved.
50
How to get started on your Graph Journey?

51. Neo4j, Inc. All rights reserved 2021

52. Neo4j, Inc. All rights reserved 2021

53. © 2023 Neo4j, Inc. All rights reserved.
53
And now in APOC too…

54. © 2023 Neo4j, Inc. All rights reserved.
54
Some examples…

55. © 2023 Neo4j, Inc. All rights reserved.
55

56. © 2023 Neo4j, Inc. All rights reserved.
56

57. © 2023 Neo4j, Inc. All rights reserved.
© 2023 Neo4j, Inc. All rights reserved.
Thank you!
Contact us at
sales@neo4j.com