Deloitte & Red Cross: Talk to your data with Knowledge-enriched Generative AI

Talk to your data with
Generative AI and
Knowledge Graphs
NEO4J GRAPH SUMMIT BENELUX
MARCH 2024

© 2024 Deloitte The Netherlands
Nearly 40% of organizations cite difficulty validating and
evaluating results, employee hesitancy to trust
recommendations, and ethical considerations and biases in
generated content1
1 Enterprise Strategy Group, Selecting a Database for Generative AI in the Enterprise , February 2024

Two applications of knowledge-enriched generative AI will be presented
Agenda
3
Fenna ten Haaf
Senior Consultant, Responsible Data & Analytics
Deloitte Risk Advisory, the Netherlands
Axel Blom
Consultant, Responsible Data & Analytics
Deloitte Risk Advisory, the Netherlands
Chatbot proof-of-concept for Red Cross volunteers
Deloitte Data Insights Monitor: GenAI Features

Red Cross Chatbot

1. Project Goal and Scope
5

Vision
A smart use of data will positively impact faster & more
(cost-)effective humanitarian aid
Products
Improve speed, quality and cost-effectiveness of
humanitarian aid by using and creating data & digital
products
Focus
• Disaster Preparedness
• Predictive Impact Analytics
• Emergency Data Support
• Direct Digital Aid
About 510
6

Introduction of the Use Case
The conflict in Ukraine forced 8 million people to
flee to European countries. Around 100.000
refugees settled in the Netherlands.
The no. of volunteers available to support
operations of the Netherlands Red Cross in
response to the Ukraine crisis has decreased
significantly, so Initiatives of the Netherlands
Red Cross are under pressure.
To enable the Netherlands Red Cross to continue
supporting Ukrainian refugees – with stable
demand but decreasing volunteers – 510 wants
support in developing a Proof-of-Concept
Chatbot to reduce the “burden” on volunteers
and remove barriers for refugees to ask for help.
SITUATION COMPLICATION SOLUTION
7

Developing a chatbot using advanced techniques as an innovative way to improve humanitarian aid
Project Goal and Scope
Goal
Develop a Proof-of-Concept Generative AI chatbot that can answer questions of Ukrainian refugees in the Netherlands,
using the existing Helpful Information App as a data source
8
GENERATIVE AI: LARGE LANGUAGE MODEL
Generate human-like text based on prompts
Provide answers to a wide range of questions
No training or finetuning of model needed
KNOWLEDGE GRAPH
Store information and its relationships to other data
Link data from different sources together, to make it searchable
Extract reliable and context-aware answers
COMBINED MODEL: Knowledge Graph-Enhanced Chatbot

2. Architecture design
9

Architecture of the Integrated KG Pipeline
10
Generate answer
Answer
Vector
Set of sim
ilar
vectors
as
context
User
LLM
Vector database
Smart search
Q
uestion
Question

11
Generate answer
Vector
Set of sim
ilar
vectors
as
context
User
LLM
Cypher LLM
Vector database
Smart search
Q
uestion
Question
Q
uestion
Answer

12
Graph Query
Generate answer
Vector
Set of sim
ilar
vectors
as
context
User
Knowledge Graph
LLM
Cypher LLM
Vector database
Smart search
Q
uestion
Question
Q
uestion
Answer

13
Graph Query
Generate answer
Vector
Set of sim
ilar
vectors
as
context
Relevant
results
as
context
User
Knowledge Graph
LLM
Cypher LLM
Vector database
Smart search
Q
uestion
Question
Q
uestion
Knowedge-enriched answer

3. Results
14

© 2024 Deloitte The Netherlands 15
Question
Which shelters are wheelchair friendly?
An example where KG context improves the Vector DB chatbot answer
Example Question
According to Document[N127], the emergency shelter location at Max
Euwelaan 1 in Rotterdam is not wheelchair friendly. However, the
document also states that if someone with a physical disability that
makes it impossible for them to use the shower comes to this location,
everything possible will be done to get them to a location where those
facilities are available. There is no information in the given documents
about other shelters being wheelchair friendly.

Question
Example Question

Question
Example Question
"The Utrecht Public Shelter, Emergency Shelter Haarlem, Emergency
Shelter Rotterdam, and Amsterdam Public Shelter are all wheelchair
friendly. However, please note that the showers at the registration
location in Rotterdam are unfortunately not wheelchair friendly.”

Question
What are the websites of all emergency shelters in Netherlands?
Example Question
"I don't have the right information to answer your question. However, I
can tell you that Document[N129] lists some municipal shelter locations
where the Red Cross is present in the Netherlands, but it does not
provide the websites of these locations. Additionally, Document[N127]
provides information about an emergency shelter location in Rotterdam,
but it does not provide a comprehensive list of all emergency shelters in
the Netherlands."

Question
Example Question
the Netherlands."

Question
Example Question
the Netherlands."
"The websites of emergency shelters in Netherlands are:
Emergency Shelter Haarlem:
https://haarlem.nl/yakscho-vi-priikhali-z-ukraini-oekraine#accordion-pa
nel-33822-1n-
Emergency Shelter Eindhoven:
https://www.eindhoven.nl/stad-en-wonen/hulp-aan-vluchtelingen-uit-o
ekrainen
Emergency Shelter Rotterdam:
https://www.rotterdam.nl/wonen-leven/vluchtelingen-oekraine/
Emergency Shelter Den Haag:
https://www.denhaag.nl/en/municipality-of-the-hague/ukraine.htm “

Question
What to do if I don’t like my host family?
An example where KG context is empty
Example Question
["I don't like rules of host family Yes, I understand. Every family has its own rules
and borders. There are some things you can like or not like. If it doesn't have a
direct impact on your life or health, you can't change it in somebodies' family. In
the Netherlands people respect each other and always give freedom of choice.
So, each of us can choose whatever he likes if it doesn't harm other people.",
'My host has different lifestyle and it doesn't look good to me Yes, I understand
that some things can be strange for you. Try to talk to your host. Dutch people
are very honest and they will rather discuss things and fix the problem, than be
silent about things they don't like. If you feel uncomfortable in some way –
explain it. Try to use phrases as: "I feel...", "I don't understand...", "I would like
you to help me with...". So, in such a way you express you own feelings and
don't blame somebody else.’,
Etc…. ]
Vector DB Context: KG Context:
[]

Question
What to do if I don’t like my host family?
An example where KG context is empty
Example Question
“If you do not like your host family, it is recommended to talk to them and express your
feelings. Dutch people are known for being honest and willing to discuss and fix problems.
You can use phrases such as "I feel...", "I don't understand...", "I would like you to help me
with..." to express your own feelings without blaming anyone. If you do not feel safe at
your host family, you can go to an emergency shelter of the municipalities. If you cannot
eat the same food as your host, you can discuss it with them and find a balance, such as
cooking some meals for yourself or using your social money to buy products for your own
use. If you are already living with a host family and do not want to leave, there is no
obligation to do so.”

4. Evaluations and conclusions
23

Sometimes, if the question is
formed in a complex structure or
what’s asked in the question is not
added to any nodes or properties
KG returns an empty context
In most cases, KG improves
performance if the question
requires a hop within multiple
nodes by providing extra context
Integrated KG pipeline can
• Take in both contexts and match
the correct context with the
related question
• Select the related context from
the graph to answer the question
without returning all the
information in the graph
Overall Evaluation of the Integrated Pipeline
24

Ragas Evaluation Metrics
25
1 2
VECTOR DB PIPELINE KG & VECTOR DB INTEGRATED
PIPELINE
Faithfulness 0,88 0,92
Context Precision 0,20 0,50
Context Recall 0,89 0,77 The context recall is lower. This can be explained
by the fact that the integrated llm enriches the
answer more often with structural information
from the knowledge graph.
The Integrated KG pipeline performs
better than the Vector DB pipeline on
both Faithfulness and Context Precision
metrics.

Relationships should be complete
Challenges – Knowledge Graph Design

The LLM does not know the property values for each node

Property and class names should be intuitive

Difficult to model ‘vague’ ideas and conditional statements

Time and expertise needed from the client
Difficult to model ‘vague’ ideas and conditional statements

Challenges – Prompt Engineering

The Integrated KG pipeline has been shown to add value to the provided answers, especially in the case when a
question applies to multiple entities, and when a question is not in the provided dataset
Conclusions and Future Work
IMPLEMENTED
IMPLEMENTED
• Single-turn chatbot
• End-to-end question answering pipeline
• Modular setup and customizable
• Evaluated on provided dataset
• Testing module
Vector DB pipeline Integrated KG pipeline
• Manually filled in CSV files that represent nodes, properties, and
relationships in the graph
• A KG parser that turns the CSV files into graph format
• A graph querying LLM
• An integrated LLM that takes KG and Vector DB contexts as input
FUTURE WORK
• Multi-turn / Conversational chatbot
• Evaluate using human input
• Improved testing
• Deploying and embedding into website
• Stability of graph queries generated
• User interface
• Automating the graph formation using entity recognition
• Trying out other integration options
32

Data Insights
Monitor

Data Insights Monitor
https://www2.deloitte.com/nl/nl/pages/risk/solutions/data-insights-monitor.html
The Data Insights Monitor (DIM) is a flexible, user-friendly
overlay for advanced analytics built on graph technology
The non-invasive platform is built on top of existing data
infrastructure and does not require any change in existing
infrastructure
The newly improved version of DIM contains GenAI
functionality to provide natural language to advanced
analytics functionality

Quickly build complex user interactions by combining recipes for a ﬂexible, user-friendly platform
Data Ingredients and Recipes

Control and guide LLMs using recipes
Existing recipes work as guidelines to control the queries generated by LLMs
Domain of
Knowledge
Domain of
Queries
Prompts

Use the Synergy of unstructured vector-based search combined with structured graph queries
Deﬁne and customize diﬀerent integration pipelines to combine vector search with graph power
Graph Vector
Integrated
Graph
Vector
Integrated
Vector
Graph
Integrated

Choose and Combine On-premise and Cloud-based LLM Models
Eﬃciency and cost-saving by applying the right set of LLMs for the right tasks

Switch between natural language queries and recipes
Data points returned as results of a natural language query can be collected for further analysis
Switch context

Switch between natural language queries and recipes
Data points returned as results of a natural language query can be collected for further analysis

Links & further information
41

Connections in Data are as valuable as the Data itself…
Knowledge-enriched AI & Analytics
More information on
Data Insights Monitor
Whitepaper
Knowledge-enriched
GenAI

Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited (“DTTL”), its global network of member firms, and their related entities (collectively, the “Deloitte organization”). DTTL (also
referred to as “Deloitte Global”) and each of its member firms and related entities are legally separate and independent entities, which cannot obligate or bind each other in respect of third parties.
DTTL and each DTTL member firm and related entity is liable only for its own acts and omissions, and not those of each other. DTTL does not provide services to clients. Please see
www.deloitte.com/about to learn more.
Deloitte provides industry-leading audit and assurance, tax and legal, consulting, financial advisory, and risk advisory services to nearly 90% of the Fortune Global 500® and thousands of private
companies. Our professionals deliver measurable and lasting results that help reinforce public trust in capital markets, enable clients to transform and thrive, and lead the way toward a stronger
economy, a more equitable society and a sustainable world. Building on its 175-plus year history, Deloitte spans more than 150 countries and territories. Learn how Deloitte’s more than 415,000
people worldwide make an impact that matters at www.deloitte.com.
This communication contains general information only, and none of DTTL, its global network of member firms or their related entities is, by means of this communication, rendering professional
advice or services. Before making any decision or taking any action that may affect your finances or your business, you should consult a qualified professional adviser. No entity in the Deloitte
organization shall be responsible for any loss whatsoever sustained by any person who relies on this communication.
© 2024. For information, contact Deloitte Netherlands. 43

Deloitte & Red Cross: Talk to your data with Knowledge-enriched Generative AI

More Related Content

What's hot

More from Neo4j

Recently uploaded

Deloitte & Red Cross: Talk to your data with Knowledge-enriched Generative AI