The document discusses the integration of large language models (LLMs) with knowledge graphs (KGs) to enhance knowledge representation and reasoning capabilities. It explores various frameworks for LLM-KG fusion, including embedding, completion, construction, KG-to-text generation, and question answering, while also addressing the limitations of baseline retrieval-augmented generation (RAG). Additionally, the document presents a case study comparing traditional RAG with GraphRAG in handling complex queries related to the term 'Novorossiya,' highlighting the advantages of GraphRAG in synthesizing insights from disparate information.

1. NLP IL Journal Club Online
GraphRAG is All You need?
LLM & Knowledge Graph
Guy Korland, PhD.
CEO & Co-Founder

2. About Me
● Married+3 (45)
● Academic
○ MSc. - Technion - Ad-Hoc Networks
○ PhD. - TAU - Multi-Core programming
● >20 years dealing with Databases
○ GigaSpaces (TupleSpace, KV, Document)
○ Redis (KV, Document, TimeSeries, Graph, Search, Vector)
○ FalkorDB (Graph)
● I’m not an NLP or LLM expert…..

3. FalkorDB Core Technology
An ultra-low latency Graph Database
FalkorDB

4. FalkorDB - The Internals
Based on GraphBLAS - (Prof. Tim Davis from Texas A&M)

5. LLM: Fastest growing Market → New Opportunities!
2024

6. Unifying Large Language
Models and Knowledge
Graphs: A Roadmap
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, Xindong Wu
V3 - last revised 25 Jan 2024

8. Knowledge Graphs

9. Knowledge Graphs

10. KGs & LLMs

11. Roadmap

12. Synergized LLMs + KG
1. Knowledge representation
2. Reasoning

13. Synergized LLMs + KG - Knowledge representation
Synergized knowledge
representation by additional
KG fusion modules.

14. Synergized LLMs + KG - Reasoning
The framework of LLM-KG Fusion Reasoning

15. LLM-augmented KG
1. Embedding
2. Completion
3. Construction
4. KG-to-text Generation
5. Question answering

16. LLM-augmented KG - Embedding
LLMs as text encoder for knowledge
graph embedding (KGE)

17. LLM-augmented KG - Embedding
LLMs for joint text and
knowledge graph embedding

18. LLM-augmented KG - Completion
The general framework of adopting
LLMs as encoders (PaE) for KG
Completion

19. LLM-augmented KG - Construction
● Entity Discovery
○ Named Entity Recognition
(NER)
○ Entity Typing (ET)
○ Entity Linking (EL)
○ Within-document CR
○ Cross-document CR
● Relation Extraction (RE)
○ Sentence-level RE
○ Document-level RE (DocRE)
○ End-to-End KG Construction

20. LLM-augmented KG - KG-to-text Generation
The general framework of KG-to-text generation.

21. LLM-augmented KG - Question answering
The general framework of applying LLMs for
knowledge graph question answering (KGQA).

22. KG-enhanced LLMs
1. Pre-training
2. Inference
3. Interpretability

23. KG-enhanced - LLM - Pre-Training
Injecting KG information
into LLMs training
objective via
text-knowledge alignment
loss.
Where h denotes the
hidden representation
generated by LLMs.

24. KG-enhanced - LLM - Pre-Training
Injecting KG information into LLMs
inputs using graph structure.

25. KG-enhanced - LLM - Inference
Retrieving external knowledge to enhance the LLM generation

26. KG-enhanced - LLM - Interpretability
● Probing
● Analysis

27. RAG
Retrieval Augmented Generation

28. Retrieval Augmented Generation (RAG)
User question
Context
Answer

29. RAG
Vector Similarity Search

30. Ingestion - Embedding
Text Embeddings
1.6, 0.42, 0.8, 1.04, 0.2
0.24, 1.3, 1.1, 0.6, 0.3
0.2, 0.81, 1.01, 0.3, 1.4
unstructured

31. Query - Embedding
1.2, 0.3, 1.2, 0.73, 0.4
User question
Embeddings

32. Query - Retrieval
1.2, 0.3, 1.2, 0.73, 0.4
User question
Semantic search
K nearest neighbor
Embeddings

33. Query - Augmentation
1.2, 0.3, 1.2, 0.73, 0.4
User question
Semantic search
K nearest neighbor
Context
Embeddings

34. Query - Response
User question Answer
Context

35. Vector Search RAG
1. Easy!
2. Plenty of vendors
3. Well documented & rich ecosystem
4. Hybrid - Semantic and Syntactic search

36. Issues / Shortcomings
Vector Similarity Search

37. 1. Naive
Q: “Fetch the latest 5 published articles”

38. 1. Naive
Q: “Fetch the latest 5 published articles”
Q: “Please recommend painkillers
which wouldn’t conflict with
the current meds patient X is taking.”

39. 2. Unaware of the domain
1. SpO2: 95%
2. PaO2: 80 mmHg
3. PaCO2: 40 mmHg
4. pH: 7.35
5. HCO3: 22
6. HR: 88
7. BP: 100 mmHg
8. RR: 15
Q: “Which patient might be at risk?”

40. 3. Can’t see the entire dataset
Q: “How many employers does company X has?”
Q: “Which article got the most reviews?”
Q: “Will a change in module A effect module Z?:

41. Q: “What’s Napoleon’s surname?”
4. Large token overhead
French Empire as
Emperor of the
French from 1804
until 1814, and
briefly again in
1815. His political
and cultural legacy
endures as a
celebrated and
controversial
Napoleon
Bonaparte (born
Napoleone di
Buonaparte;
[1][b]
15 August 1769 –
5 May 1821), later
known by his
regnal name
Napoleon I, was
considered one of
the greatest military
commanders in
history and his wars
and campaigns are
still studied at
military schools
worldwide.
However, historians
still debate the
Bonaparte

42. There’s a real problem relaying on this approach
● In the “best” scenario the system will not provide an answer to a question,
saying I don’t know.
● Far worse the system will come up with half truths, providing us with false
information which we might act on.
Vector conclusion

43. GraphRAG: Unlocking
LLM discovery on
narrative private data
Jonathan Larson , Steven Truitt. Microsoft Feb 2024

44. The problem - Baseline RAG
● Struggles to connect the dots. This happens when answering a question
requires traversing disparate pieces of information through their shared
attributes in order to provide new synthesized insights.
● Baseline RAG performs poorly when being asked to holistically understand
summarized semantic concepts over large data collections or even singular large
documents.

45. Benchmark: Baseline RAG vs GraphRAG
● the Violent Incident Information from News Articles (VIINA) dataset.
● Was chosen due to its complexity and the presence of differing opinions and
partial information.
● It is a messy real-world test case that was recent enough not to be included in
the LLM base model’s training.
● News articles from both Russian and Ukrainian news sources for the month of
June 2023, translated into English,

46. Benchmark: Baseline RAG vs GraphRAG
Query: “What is Novorossiya?”
Baseline RAG GraphRAG
The term “Novorossiya” is not mentioned in the provided
context. However, to provide some information, Novorossiya,
or New Russia, is a historical term referring to a region in the
south of Ukraine that was conquered by the Russian Empire
in the 18th century. In recent years, the term has been
revived by Russian-backed separatists in Eastern Ukraine to
refer to the territories they claim, but it is not officially
recognized as a separate entity by international bodies.
Novorossiya, also known as New Russia, is a term with
historical and contemporary significance. Historically, it
referred to a region in the southern part of the Russian
Empire that is now part of Ukraine. In the contemporary
context, it has been used by pro-Russian separatists to refer
to certain areas of Ukraine [Data: Entities (6494, 912, 6493)].
The term Novorossiya has been used in the context of a
political movement in Ukraine, particularly by Russian
nationalists. This movement has been associated with efforts
to destabilize Ukraine and create a so-called “Novorossiya”
[Data: Entities (912); Relationships (15211, 15206)]. The
term has also been used to denote a separatist state in
Ukraine, known as the Federal Republic of Novorossiya
(FRN) [Data: Entities (7690)].

47. Benchmark: Baseline RAG vs GraphRAG
Query: “What has Novorossiya done?”
Baseline RAG GraphRAG
The text does not provide specific information on what
Novorossiya has done.
Novorossiya, a political movement in Ukraine, has been
involved in a series of destructive activities, particularly
targeting various entities in Ukraine [Entities (6494, 912)]. The
movement has been linked to plans to destroy properties of several Ukrainian entities,
including Rosen, the Odessa Canning Factory, the Odessa Regional Radio Television
Transmission Center, and the National Television Company of Ukraine [Relationships
(15207, 15208, 15209, 15210)]. In addition to these, Novorossiya has also been
associated with plans to destroy properties of Roshen, the Odessa Conservatory
Factory, the Odesa Regional Radio Television Transmission Center, and the National
Telecommunications Company of Ukraine [Relationships (8329, 8330, 8331, 8332)].
Furthermore, Novorossiya has been implicated in plans to blow up ATMs of PrivatBank,
the largest state bank in Ukraine [Relationships (8335, 8335)].
The Office of the General Prosecutor in Ukraine has reported on the creation of
Novorossiya, indicating the government’s awareness and potential concern over the
activities of this movement [Relationships (8264, 8264)]. The destructive activities of
Novorossiya have had significant impacts on the targeted entities and have likely
contributed to the ongoing tensions and conflicts in the region.
In summary, Novorossiya has been involved in a series of destructive activities
targeting various entities in Ukraine. These activities have had significant impacts on
the targeted entities and have likely contributed to the ongoing tensions and conflicts in
the region. The Ukrainian government, through the Office of the General Prosecutor,
has acknowledged the existence and activities of Novorossiya, indicating a level of
concern over the movement’s actions.

48. Behind the scenes: Baseline RAG
The source documents inserted into
the context window none of the text
segments discuss Novorossiya
→resulting in this failure.

49. Behind the scenes: GraphRAG
Discovered an entity in the query,
Novorossiya.
→ Allows the LLM to ground itself in
the graph
→A superior answer that contains
provenance through links to the
original supporting text.

50. Benchmark: Baseline RAG vs GraphRAG
Whole dataset reasoning
Query: “What are the top 5 themes in the data?“
Baseline RAG GraphRAG
1. Improving the quality of life in cities and urban
development.
2. The meeting between Vladimir Putin and the President of
the United Arab Emirates, Mohammed bin Zayed Al Nahyan.
3. The National Rating of the Investment Climate in the
Regions of Russia.
4. The state of the Russian economy and the growth of
domestic brands.
5. The discussion on the meaning of life and the role of
science in understanding it.
The data presents a wide range of themes, but the top five
most prevalent themes can be identified as follows:
1. Conflict and Military Activity: A significant portion of the
data revolves around the ongoing conflict in Ukraine, with
numerous reports detailing military activities, occupations,
and attacks by various entities. This theme includes the
actions of the Ukrainian military, Russian forces, and various
other military units and entities [Data: Reports (513, 241,
151, 493, 343, +more)].
2. Political and Governmental Entities: Many reports focus
on political figures and governmental entities, highlighting
their roles, relationships, and actions in the context of the
current situation. This includes entities like the Kremlin, the
President of Ukraine, the US State Department, and various
regional administrations [Data: Reports (98, 493, 265, 129,
196, +more)].

51. Benchmark: Baseline RAG vs GraphRAG
Faithfulness
● SelfCheckGPT to perform an absolute
measurement of faithfulness to help ensure
factual
● GraphRAG achieves a similar level of
faithfulness to baseline RAG.
● Developing an evaluation framework to
measure performance on the class of
problems.

53. Evaluation

54. Q&A
Q: “Who fought in the most recent match?”
Graph: In the most recent match, Derek Brunson fought against Kevin Holland.
(20 March 2021)
Vector: The most recent match was fought between Markus Perez and Dalcha
Lungiambula.
(20 January 2021)

55. Q: “Which fighter holds the fastest win?”
Graph: The fighter who holds the fastest win is Jorge Masvidal, with a win in just
5 second
Vector:
The fighter with the fastest win, based on the provided context, is Andre Winner,
who won by KO/TKO against Rolando Delgado. The fight was concluded at 202
seconds into the first round.
Q&A

56. Q: “Which fight had a clear underdog who’ve yet won the match?”
Graph: In a fight where the victory was seemingly against the odds, Dan Severn
emerged as the clear underdog yet managed to secure a win against Ken Shamrock.
This was indicated by the comparison of their SLpM and TD_Avg.
Q&A

57. Q: “Which fight had a clear underdog who’ve yet won the match?”
Vector: Based on the information provided, identifying the fight with a clear underdog who then won the match requires assessing disparities in performance metrics or fight
outcomes that might suggest an unexpected victory. However, without explicit pre-fight odds or narratives indicating an underdog status, determining an "underdog" victory
from statistical data alone can be challenging, especially when fights end in decisions that don't necessarily reflect a dominant performance against the odds.
Looking at standout performances against expectations or significant performance turnarounds within the provided context might help, yet the data gives us detailed stats rather
than pre-fight expectations. One classic sign of an underdog victory might be a lower overall statistical expectation ending in a KO/TKO, suggesting a turnaround.
One instance noted is the fight between Belal Muhammad and Vicente Luque, where Luque won by KO/TKO in the first round at 79 seconds despite the fight's short duration
giving limited insight into the performance metrics before the finishing moment. However, labeling Luque as an "underdog" without pre-fight odds or broader fight context is
speculative.
The provided data focuses on fight statistics like significant strikes, control time, and method of victory without explicitly indicating favorites or underdogs. In Mixed Martial Arts
(MMA), unexpected victories often stand out due to knockout power or submission skills that defy statistical trends, but these aspects aren't directly reflected in the metrics
provided.
Without clear indicators of expectations versus outcomes, or a narrative context surrounding each fight, it's difficult to definitively pinpoint a fight featuring a clear underdog who
won.
Q&A

58. Q: “List fighters who had a trilogy match”
Graph: Matt Hughes vs. Georges St-Pierre
Daniel Cormier vs. Stipe Miocic
Junior Dos Santos vs. Cain Velasquez
Q&A

59. Q: “List fighters who had a trilogy match”
Vector: Based on the given context, identifying specific fighters who have had a trilogy match (three matches against the same opponent) within the
dataset provided is not directly possible. The context supplied is a list of fight statistics for various matchups but does not detail multiple matches
between the same pairs of fighters or explicitly label any series of fights as part of a trilogy.
To determine trilogy matches, one would typically look for repeated occurrences where the same pair of `B_fighter` and `R_fighter` appear against each
other across different dates. However, this dataset does not reveal any such repeated pairings from the given entries, or at least, such analysis would
require additional processing to identify any repeating matchups, assuming the data spans all fights in such trilogies. Moreover, the context provided lists
each fight as a separate entry without linking multiple entries to indicate ongoing rivalries or sequential matchups.
In the world of mixed martial arts (MMA), trilogies are often highlighted and marketed due to the evolving nature of the competition between two fighters.
They typically occur after both fighters have one win against each other, leading to a third match to break the tie or due to the competitive and
entertaining nature of their previous encounters that demand a rematch.
Without further data linking fights into series or direct mention of such trilogies in the supplied context, identifying fighters with trilogy matches within
this dataset is not feasible.
Q&A

60. Try it
https://github.com/falkorDB/UFC

61. Chat history
Options:
● Limited history
● Vector Database
● Knowledge Graph

62. Thank you!