EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

www.earley.com
WEBINAR
WEBINAR
The Key to Context:
Prompt Engineering and Knowledge
Engineering
SETH EARLEY
CEO & FOUNDER
EARLEY INFORMATION
SCIENCE
Media Sponsor
MIKE DOANE
DIRECTOR, CONTENT
DELIVERY
CIGNA HEALTHCARE
SANJAY MEHTA
PRINCIPAL SOLUTION
ARCHITECT
EARLEY INFORMATION
SCIENCE

www.earley.com
Today’s Speakers
Seth@earley.com
https://www.linkedin.com/in/sethearley/
2
Mike Doane
Director, Content Delivery
Cigna Healthcare
michael.doane@evernorth.com
www.linkedin.com/in/mikedoane/
Seth Earley
Founder & CEO
Earley Information
Science
Sanjay Mehta
Principal Solution Architect
Earley Information Science
Sanjay.mehta@earley.com
https://www.linkedin.com/in/sanjaymehta/
“I do not know of any books that have such
useful and detailed advice on the relationship
between data and successful conversational AI
systems.”
—Tom Davenport, President’s Distinguished
Professor at Babson College, Research Fellow at
MIT Initiative on the Digital Economy, and author of
Only Humans Need Apply and The AI Advantage

www.earley.com
Before We Get Started
WE ARE RECORDING SESSION WILL BE
50 MINUTES PLUS
10 MINUTES FOR
Q&A
YOUR INPUT IS
VALUED
Link to recording &
slides will be sent by
email after the webinar
Use the Q&A box to
submit questions
Participate in the polls
during the webinar
Feedback survey
afterward (~1.5 minutes)
Thank you to our media partners : CMSWire
3

www.earley.com
About Earley Information Science
4
Proven methodologies to organize information and data.
SELL MORE
PRODUCT
SERVICE
CUSTOMERS
EFFICIENTLY
INNOVATE
FASTER
1994
YEAR FOUNDED.
Boston
HEADQUARTERED.
50+
SPECIALISTS & GROWING.

www.earley.com
Poll
5
1. Not on the radar
2. Planning stages for Gen AI
3. Controlled experiments using Gen AI
4. Gen AI usage is currently banned
5. Implemented PoC’s (internal or externally facing)
6. Gen AI applications deployed
7. None of the above
Where are you on your Gen AI journey?

www.earley.com
Agenda
6
There’s No AI Without IA
Knowledge Engineering
• Taxonomies, Ontologies and Knowledge Graphs
Knowledge Graphs and LLMs
• Content and Metadata
Prompts as Metadata
• Deriving Prompts from Use Cases
Building Standardization through Libraries of Use Cases
Next Steps

www.earley.com
7
How is taxonomy foundational?
What is meant by ontologies being relational?
Why is integration with content management critical?
How is knowledge harvested?
What is the role of Subject Matter Experts (SME’s)?

Copyright © 2024 Earley Information Science, Inc. All Rights Reserved.
“THERE’S NO AI WITHOUT IA”
Knowledge Architecture is Needed to Support
Conversational and Cognitive Applications
8

www.earley.com
Knowledge Architecture
Knowledge engineering is a field of artificial
intelligence (AI) that tries to emulate the judgment
and behavior of a human expert in a given field.*
*https://www.techtarget.com/searchenterpriseai/definition/knowledge-engineering
Knowledge architecture consists of the design
artifacts and supporting technologies and
processes that enable a contextualized information
ecosystem.**

www.earley.com
Poll
11
1. No formal KM programs
2. Early stages of KM
3. KM is used at the departmental level
4. KM is widely deployed and operationalized
Where are you on your Knowledge Management
(KM) journey?

Navigation versus Classification
12
Classification Hierarchies
• Allows for definition of “is-ness” (what is this thing?) and
“about-ness” (what is it about then that helps me tell them
apart?)
• Classification drives dynamic navigation via facets which
leverage is-ness and about-ness (What is this? A sweater. Tell
me about this sweater. It’s blue)
• Relationships between classification hierarchies defines the
ontology (Products for Processes, Processes for Industry, etc.)
Navigational Hierarchies
• What most people think of when they
hear the term “taxonomy”
• Core structure of organizing principles for
a collection of information
• Static navigational hierarchies
(navigational taxonomies) is a dated
approach for any but most rudimentary
sites
• Dynamically driven by classification
hierarchies
We are not talking about
navigational hierarchies
(sometimes called “business
taxonomies”) due to lack of
adherence to classification
rules
Taxonomy is not the same as navigation

Taxonomies to Ontologies and Knowledge Graphs
13
Classification
Hierarchies
Define Ontologies
Ontologies, when
connected to data sources
becomes a Knowledge
Graph the knowledge
scaffolding of the
enterprise
=> Connect to Data
=>
Knowledge Graph

Knowledge Graphs
14
What is the purpose of a Knowledge Graph?
How are linguistics and analytics integrated?
Knowledge Graphs connect the LLM to
corporate knowledge and data.
They are the source of truth for the LLM so
that responses are appropriate to enterprise
information

www.earley.com
www.earley.com
Generative AI
15
Creates new, original content
Trained to learn the patterns and relationships
Learned knowledge generates new content
(Not a copy of the training data).
This means that the knowledge of the organization needs
to be referenced by the technology
*Source: ChatGPT

www.earley.com
www.earley.com
Content Management Integration
16
Content (and Knowledge) Management is needed to:
• Ensure that the chatbot has access to a wide range of relevant
information and can provide accurate and useful responses to
user queries
• Identify knowledge and expertise needed for the chatbot to
function effectively
• Design the chatbot's knowledge base and information
architecture to support user needs.

www.earley.com
www.earley.com
Integration of Knowledge Graphs with LLMs
17
How do Knowledge Graphs Enhance Domain-specific
Understanding?
How can context be improved?

www.earley.com
www.earley.com
Generation vs Retrieval
18
User query Process query
using LLM to
understand
user intent
Generate response
based on LLMs
understanding of
language patterns and
concept relationships
Retrieve response based
on querying
organizational
knowledge and content
Process response
using LLM to
provide
conversational
format
Uses publicly
available information
Uses proprietary
information and
nonpublic IP
Does not compromise
or expose IP
LLMs used to process query
and present results
Response
Corp data
sources
Vector Store
Corporate information is referenced
through the knowledge graph

www.earley.com
www.earley.com 19
Knowledge Graph as Ground Truth
Process with LLM with
question and relevant
documents
User query
Search/
retrieval
Response
Corporate information
retrieved through sources
in knowledge graph
Generated results based
on provided documents

www.earley.com
What is Context ?
20
The ability to understand who the user is and what they want.
We can break the process down of
referencing a manual:
Scenario 1 – Access
• Here is the manual
Scenario 2 – Generalized retrieval
• “Look in chapter 4”
Scenario 3 – Specificity of the answer
• Here is the specific answer to your
question from that manual
Scenario 4 – Contextualized knowledge
• Here is the specific answer from the
manual and related information based
on your exact product configuration
and context
Manuals compile knowledge for
technical support
However…
• They require study
• And it takes too long to find
answers to specific questions
from large manuals.
(RTFM – TLDR)
What is the user’s context?
How do people find information?

www.earley.com
www.earley.com Copyright © 2023 Earley Information Science, Inc. All Rights Reserved.
Customer Identity Graph
How do we
describe
context?
With
metadata.

www.earley.com
www.earley.com Copyright © 2023 Earley Information Science, Inc. All Rights Reserved.
Explicit and Implicit Customer Metadata
Where do we get
metadata?
By collecting signals
instrumented throughout
the user journey

© 2024 Earley Information Science, Inc. All Rights Reserved.
www.earley.com
www.earley.com
USER CONTEXT, PROCESS CONTEXT AND CONTENT CONTEXT
23
How do I set up my modem?
Where is the installation guide?
What does error code 50 mean?
Questions and Answers Need Context

www.earley.com
www.earley.com
Product Manual – HM 2900 Series Modem
Generative AI Reference Content Requires Context
24
Overview
Set up options
Settings Model 2960
Error code 50
Settings Model 2970
Error code 56
Installation
Troubleshooting
Hardware setup
VPN requirements
Factory settings
Technical Specifications
I need to install a modem.
Which modem model do you have?
Content type = Product Manual
Content type =
Troubleshooting
Content type = Installation
Model 2960
I am receiving an error code of 50
OK, here are the installation settings…
That error requires the following
troubleshooting steps:
What context is required?
Type of information, installation, product model, error code, etc.
Metadata provides context
Product name = HM 2900 Series
Modem
Model = 2960
Error code = 50

www.earley.com
www.earley.com
Prompt as Metadata Container
25
“I need to install a model 2960 modem, but I am receiving an
error code of 50. Please provide the troubleshooting steps to
complete my installation”
Content type = Product Manual
Content type =
Troubleshooting
Content type = Installation
What context is required?
Type of information, installation, product model, error code, etc.
Prompt metadata provides context
Product name = HM 2900 Series
Modem
Model = 2960
Error code = 50

www.earley.com
www.earley.com
Use Cases as Prompt
26
Use cases inform the knowledge architecture.
Libraries of use cases allow for testing functionality, facet
identification and baseline metric comparison
Role metadata Entity metadata

www.earley.com
www.earley.com
Implications
27
Need reference prompts representing classes of use case
Training material for how users need to ask questions
Knowledge architecture provides clues as to how to ask
questions to get content in context
Analogous to faceted retrieval – an ambiguous question
requires disambiguation through filtering

www.earley.com
www.earley.com
Types of Prompts
28
One shot – no context is needed – rote process (text
summarization, sentiment analysis, spam classification).
Few shot – Additional context provided through inputting
relevant content with the prompt.
Prompts become metadata – the role of the
customer, technical vs business, the role of the
bot “you are an expert in brand marketing”

www.earley.com
www.earley.com
Types of Prompts
29
Chain of thought prompting – Break the prompt into multiple
steps and show intermediate results.
- Different lines of thinking and compares results
- Build the output based on sequential outputs of sub
problems or steps
- Monitoring confidence and escalating for human review.

www.earley.com
www.earley.com
Prompt Metadata
30
Asking for outputs and then using knowledge to ask additional
questions.
“What questions should I ask about Generative AI and prompts?”
Understanding the Prompt's Purpose:
• What is the main goal or desired outcome of this prompt? (e.g., generate
creative text formats, translate languages, write different kinds of creative
content)
• Who is the target audience for the generated output? (e.g., general public,
specific professionals, children)
• In what context will the output be used? (e.g., marketing materials,
educational content, personal entertainment)
Crafting an Effective Prompt:
• What specific information or keywords must be included in the prompt?
• How can I phrase the prompt to ensure clarity and avoid ambiguity?
How can you describe
the output? (tone,
mood, technical
depth)
How can you describe
the audience?
What is the expected
artefact (plan, program,
campaign, contract)?
What additional
metadata (features) can
you provide?
What is the use case?

01 Role, Target, Purpose, Constraints
Providing model with macro context (ex. Industry), organizational context, brand
language, objectives (outcomes/goals), role(s) & scope, access controls (ex. guard rails)
Incorporate user specific context (ex. Preferences, history, timeframe)
02 Examples, Few Shot, Chain of Thought, Hints
Provide model with scenarios and outcomes with reasoning steps. Ex. Providing a
problem & solution, a recipe, Q&A, format, directional stimulus, insights such as
analytics.
03 Chaining, Synthetic Signals, Multi-Modal
Ask the LLM generate an output to feed to another prompt, training data, examples. Ex.
Generate application specific keywords to feed to another prompt to generate a
description. Extract labels from an image and feed to prompt to generate a name.
04 Retrieval Augmented Generation
Provide model with results from internal knowledge base, limit results to only what is
within knowledge base or content provided using retrieval approaches such as
vector, semantic, lexical, graph...
05 Signals & Evaluation
Reinforcement context, outcome analysis, historical q&a pairs, query outcome, KPI
targets.
LLM Integration With Enterprise Knowledge Base & Content

An Example
Source: Rec Prompt https://arxiv.org/html/2312.10463v1

www.earley.com
Poll
33
1. Forget leadership, its not known at any level
2. Pockets of understanding throughout the organization
3. Senior managers and leaders understand the
connection
4. KM programs are actively integrated with LLM projects
through approaches like Retrieval Augmented
Generation (RAG)
Is there awareness at the leadership level of the
connection of Knowledge Management to Gen AI?

www.earley.com
Getting Started /Next Steps
34
• Ensure adequate funding and executive support
• Define use cases
• Identify bodies of content needed to support use cases
• Gather baseline metrics for supported processes
• Build out strawman domain model
• Develop metadata structure for target use cases
• Tag content with taxonomy, ontology and reference knowledge graph
• Ingest tagged content into vector store as enriched embeddings
• Test against use cases with gold standard of responses
• Instrumentation of processes to show value and improvement
• Shampoo, Rinse, Repeat

www.earley.com
www.earley.com
www.earley.com
Are you measuring the following?
Quality and completeness of answers in knowledge base
Usage of knowledge resources when handling calls
Contributions to knowledge repositories by reps
Consistency across knowledge sources
Correlation of knowledge source quality with first call resolution, time per incident, CSAT, LTR,
and other call center metrics
Consistency and recency of taxonomies (compliant with best practices)?
Call deflection to self service
35

www.earley.com
www.earley.com
www.earley.com
KM for AI Readiness Sprint and Proof of Value Plan
A KM for AI readiness sprint will quickly identify how the organization can deploy Generative AI to address
issues and challenges that arise from knowledge capabilities that have not kept up with growth, acquisitions,
changes in the marketplace or changes in products, technologies and the competitive landscape.
The sprint consists of the following:
• Stakeholder interviews
• Education and alignment
• Current state maturity
• Review of knowledge systems and tools
• Generative AI Proof of Value (PoV) plan
https://www.earley.com/km-ai-readiness-assessment

www.earley.com
www.earley.com
www.earley.com
Maturity Model for Knowledge Management
1-Unpredictable 2-Aware 3-Competent 4-Synchronized 5-Choreographed
Core Collaboration Rudimentary, random ,
haphazard
Intentional, ineffective
knowledge capture and
codification
Practices identified, formal
harvesting and promotion of
output
Integrated into processes
with creation, access and
reuse mapped
Seamless and habitual with
collaboration processes integrated
with business needs and
downstream uses
Expertise Location Who you know Word of mouth Key skills Identified and
captured
Formal expertise directories
leveraged
Expertise derivation through text
analytics, community participation
and group interactions
Content Curation Haphazard and application
limited
Pockets of curated
content, sub optimized
across processes
Content aligned with
platform and process,
governance driven
information concierge
Workflow driven integration
with hybrid tagging and entity
extraction processes
Value added at each touch point,
core metadata flows with content,
lifecycles managed , retention
enforced
Information
Architecture
IA is Navigation, inconsistent
metadata, few standards,
poor usability
Application and
department localized
taxonomies
Classification structures
applied to support dynamic
knowledge
Multi-channel, device and
format independent cross
application architecture
Upstream supply chain and
downstream syndication w/ partner
& customer processes
Infrastructure Foundational, little to no
collaborative tools, CM
rudimentary
Foundational tools in
place but with out of the
box deployment
Knowledge harvesting
integrated with
collaboration and ruse
Expertise location,
community management,
intentional knowledge
optimization
Multiple tools mapped to detailed
requirements and use cases with
ongoing tuning and enhancement
Search Integration Search as “Random
Document Generator “
Some tuning of search
ranking algorithm with
content tagging
Integration across
structured and unstructured
systems for content in
context
Expertise mapped to search
domains and terms, reduced
e-discovery risk
Search driven integration across
platforms, knowledge in task and
process context
Governance Non existent Initial attempts lead to
fiefdoms
Repeatable, defined KM
governance
Integrated, cross functional
managed processes
Business value driven, enterprise
wide deployment
Capability
Maturity

Knowledge Management and AI Working Session Topics
Topic Overview Goals Questions to Address
Key Concepts and
Success Criteria
Generative AI and knowledge
management definitions and success
factors.
• Level set on key terms and foundational
understanding.
• How can the organization make
use of Generative AI?
• What is the role of knowledge?
Knowledge in
Context
Overview of business objectives and
use cases for knowledge application
• Orient client team to multiple knowledge
taxonomies.
• Get initial response on which are most
important (to client).
• Which business priorities must
the AI serve first?
• Which teams will be most
involved?
Current
Landscape
Definition
Map knowledge and content by people
and systems to identify how business
processes are currently supported.
• Develop more complete view of scope and
scale of systems with knowledge and
content.
• Map knowledge to process
• Develop domain model
• Where are key knowledge
leverage (usage)
points/processes?
• Where are key areas of changes
re: people and process?
PoV Planning Plan and proposal for Generative AI
Proof of Value.
• Implement knowledge base with AI powered
chat front end
• What are the costs and needed
infrastructure for execution?

www.earley.com
www.earley.com
www.earley.com
The EIS KM and AI Readiness Assessment Can Help Get You There
Through a combination of interviews, questionnaires, surveys and working sessions, the EIS KM for AI
Readiness Assessment:
1. Educates executives and stakeholders about AI technologies – capabilities and limitations
2. Evaluates business value and target use cases for Generative AI
3. Outlines success factors and metrics
4. Examines critical areas of the enterprise for Generative AI readiness:
• Business alignment and process clarity
• Knowledge and Data readiness and technology infrastructure
• Ongoing governance, decision making and success measures
5. Summarizes the current state in an executive working session designed to identify gaps, set realistic goals and
prioritize actions for a Generative AI Proof of Value (PoV)
39
The output of the EIS KM and AI Readiness Assessment
is a roadmap for deployment of a Generative AI PoV
based on corporate knowledge

www.earley.com
www.earley.com
www.earley.com 40
Need Clarity on Retrieval Augmented Generation
(RAG)? https://www.earley.com/ama-article
Great companion to
The AI Powered
Enterprise
Download
now:
or you can request a physical reprint

www.earley.com
Contact
Seth@earley.com
https://www.linkedin.com/in/sethearley/
41
Mike Doane
Director, Content Delivery
Cigna Healthcare
michael.doane@evernorth.com
www.linkedin.com/in/mikedoane/
Seth Earley
Founder & CEO
Earley Information
Science
Dave Skrobela
Client Partner
Managing Director
dave.skrobela@earley.com
Sanjay Mehta
Principal Solution Architect
Sanjay.mehta@earley.com
https://www.linkedin.com/in/sanjaymehta/
www.linkedin.com/in/skrobela/

www.earley.com
42
We Make Information More Useable, Findable, And Valuable
Earley Information Science is a professional services firm headquartered in Boston and founded in 1994. With over
50+ specialists and growing, Earley focuses on architecting and organizing data – making it more findable, usable,
and valuable.
Our proven methodologies are designed to address product data, content assets, customer data, and corporate
knowledge bases. We deliver scalable solutions to the world’s leading brands, driving measurable business results.

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Recommended

Recommended

More Related Content

Similar to EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Similar to EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx (20)

More from Earley Information Science

More from Earley Information Science (14)

Recently uploaded

Recently uploaded (20)

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Editor's Notes