SlideShare a Scribd company logo
Knowledge
engineering: from
people to machines
and back
Elena Simperl, @esimperl
Knowledge
engineering
• Building and maintaining
knowledge-based systems
• Sits between
• Software engineering: how to
build and maintain software
systems;
• AI: how to represent, learn,
and reason over knowledge
computationally
KNOWLEDGE ENGINEERING: BEFORE
• Gathering curated knowledge
from experts and encoding it into
knowledge bases.
• Mostly manual process,
supported by methodologies,
methods, tools.
• Not scalable, considerable up-
front investment.
3
Source: NeOn project, 2010
KNOWLEDGE ENGINEERING: TODAY
• Large knowledge bases, drawn from
heterogeneous data, automatic process w/
human-in-the-loop.
• Mainstream adoption in search, intelligent
assistants, digital twins, supply chain
management, legal compliance etc.
• W/ access to data and (off-the-shelf) AI
capabilities, costs are a fraction from what they
were decades ago.
• New knowledge engineering experiences:
prompting, conversational tools, co-auditing
automatically generated content
4
Source: Fathallah et al., 2024
My KE
journey
5/28/2024 5
Standard
knowledge
engineering
process during
my PhD
6
Source: me, 2007
Case studies,
guidance,
templates to
reuse ontologies
effectively
7
Predicting costs,
understanding
the most
resource-
intensive part of
the process
8
Two ways forward
Automation Involve more people
9
Two challenges
Participation
Data quality
10
Participation:
aligning tasks
to people’s
motivations,
designing
incentives
11
People implicitly
contribute to the
knowledge base
while going about
their regular work.
Benefits are
highlighted
Challenges: not suitable
for every use case,
shallow semantics
12
People play
games. Games
data is turned
into
computational
knowledge
Challenges: design,
attracting and
retaining players,
costs
13
People are part of
online communities
that build
knowledge bases
Challenges: uneven
participation, data quality
14
What if
automation
were an
option?
15
Human computation Large language models
Knowledge
integrity in
Wikidata
Wikidata is a widely used secondary source of knowledge
Data is trusted when it is backed by its primary source
How references are encoded in Wikidata
Are
references
easy to
navigate,
authoritative,
relevant?
Yes, but some variations
across languages
5/28/2024 17
English, Swedish, Japanese, Spanish, Dutch, Portuguese
Does the
source
support
the claim?
Yes, but challenging task to
automate because of
heterogeneity and
multimodality of evidence
18
(Chandler Fashion Center, directions, Highway 101 & Highway 202)
«■ Not logged in Talk Contributions Create account Login
voyage
Page Discussion Read Edit View history | Search Wikivoyage
This is an old revision of this page, as edited by SelfieCity (talk | contribs) at 20:58,13 October 2019 (page banner). (diff)
•— Older revision | Latest revision (diff) | Newer revision —» (diff)
Main page
Travel destinations
Star articles What's
Nearby?
Tourist office
Random page
North America > United States of America >
Chandler
Southwest (United States of America) > Arizona > Greater Phoenix > Chandler (Arizona)
Travellers' pub
Recent changes
Community portal
Maintenance panel
Policies
Help
Interlingual lounge
Donate
Chandler is a city in Arizona, and a medium-sized suburb of Phoenix with over 240,000 residents. It is a delightful place to visit.
Get in
Related changes
Upload file
By plane
Phoenix Sky Harbor International Airport (PHXIATA) +1 602 275-4958 [1]i? is the main air gateway to Arizona. It is in East Phoenix 3 miles from downtown. All major U.S. carriers serve Phoenix
Special ।
Perm am
Page inft
Cite this
F rirt s r
* Q Chandler Fashion Center^ (Chandler Mali), 3111 W Chandler Blvd (Highway 101 & Highway 202), W +1 480-812-8488. M-Sa 10AM-9PM, Su 11AM-6PM; restaurant and dept store hours vary An upscale shopping mall with various
department stores, (updated Sep 2018)
• H me Shoppes at Casa Palomas’, 7131 West Ray Road (Highway 10 & Ray). Upscale outdoor shopping and dining center, (updated Sep 201 B>
•BPhoenix Premium Outletss’, 4976 Premium Outlet Way (Highway 10 & Highway 202), +1-480-639-1766. A shopping destination for locals and visitors looking for upscale shopping in a casual, family-friendly atmosphere.
(updated Sep 2018)
In other projects Get around
Wikimedia Commons
Wikipedia
Car and bus are the easiest ways to get around Chandler Rideshare services Uber and Lytt also operate in the city. Google's Waymo autonomous car service is also available in parts of the city.
Datasets
WDV:
verbalisation
WTR: textual
references
WTR
416 triples, 3 classes (supports,
refutes, not-enough-information),
free-text rationale to manage
diversity of reference layouts
Complex task, 87.5% accuracy when
explicit textual evidence present
27
What’s
next
28
5/28/2024 30
Dagstuhl
seminar
09/22
Hackathon
08/23
5/28/2024 33
How do
knowledge
engineers use
LLMs
Ethnographic observations of the
hackathon
Documentary analysis of documents
collected during the hackathon
Semi-structured interviews with hackathon
participants
Findings
Challenges around lack of datasets and
evaluation protocols
Guidance and best practices needed on
knowledge prompting and auditing LLM
outputs
Limited awareness of responsible AI
concerns like biases, fairness
35
LLMs and
requirements
engineering
AI assistants that
automate tasks,
generate
documentation,
remove technical
jargon
84.1% of knowledge engineers lack access to
specialised tools
66.1% lack access to guidelines
Manual process, using text editors and spreadsheets
[1] Monfardini GK, Salamon JS, Barcellos MP. Use of Competency Questions
in Ontology Engineering: A Survey. Int. Conf. on Conceptual Modeling 2023
Example: OntoChat
https://huggingface.co/spaces/b289zhan/OntoChat
37
User research: mock-ups
Using LLMs and
knowledge
graphs to
reverse
engineer
competency
questions
39
Explanations
5/28/2024 40
Cards for KGs and KG embeddings
Final thoughts
• Knowledge engineering is transforming. We need
• Methodologies w/ stronger focus on hybrid workflows, new roles (prompting), new
skills (co-auditing, responsible AI)
• Viable AI copilots, multimodal support, scalable to real knowledge graphs
• Datasets, benchmarks, and evaluation protocols to improve the technical
performance of LLMs in knowledge engineering tasks (e.g. knowledge prompts,
multi-turn dialog, safety, user stories, CQs, documentation etc.)
• Campaigns to share and reuse data where possible to reduce data-related costs
(prompting collectives)
• Guidance, best practices to use AI effectively
• Mechanisms to demonstrate AI-co-generated knowledge graphs can be trusted
42
Where is
the web in
all this?
43
This talk presented work by: Gabriel Amaral,Jacopo
de Berardinis, Fiorela Ciroku, Jonathan Hare, Lucie
Kaffee, Jongmo Kim, Elisavet Koutsiana,Albert
Meroño-Peñuela, Michelle Nwachukwu,Alessandro
Piscopo,Valentina Presutti, Odinaldo Rodrigues,
Johanna Walker, Bohui Zhang,Yihang Zhao
Thank you also to participants of the Dagstuhl
seminar, hackathon, and ESWC special track.
Elena Simperl
@esimperl
44

More Related Content

Similar to Knowledge engineering: from people to machines and back

Data_Mining_for_Software_Engineering.pdf
Data_Mining_for_Software_Engineering.pdfData_Mining_for_Software_Engineering.pdf
Data_Mining_for_Software_Engineering.pdf
assadabbas22
 
Technology Motivators and Usage in Non-Profit Arts Organizations
Technology Motivators and Usage in Non-Profit Arts OrganizationsTechnology Motivators and Usage in Non-Profit Arts Organizations
Technology Motivators and Usage in Non-Profit Arts Organizations
CAMT
 
siegelCV.doc.doc
siegelCV.doc.docsiegelCV.doc.doc
siegelCV.doc.doc
butest
 
siegelCV.doc.doc
siegelCV.doc.docsiegelCV.doc.doc
siegelCV.doc.doc
butest
 
Bridging Gaps and Broadening Participation in Today's and Future Research Com...
Bridging Gaps and Broadening Participation inToday's and Future Research Com...Bridging Gaps and Broadening Participation inToday's and Future Research Com...
Bridging Gaps and Broadening Participation in Today's and Future Research Com...
Sandra Gesing
 
Chapter 1 Introduction to Datascience (1).pptx
Chapter 1 Introduction to Datascience (1).pptxChapter 1 Introduction to Datascience (1).pptx
Chapter 1 Introduction to Datascience (1).pptx
kiitlabsbsc
 
ICSE 2016 - Opening and Awards I
ICSE 2016 - Opening and Awards IICSE 2016 - Opening and Awards I
ICSE 2016 - Opening and Awards I
sonal-mahajan
 
[Keynote] predictive technologies and the prediction of technology - Bob Will...
[Keynote] predictive technologies and the prediction of technology - Bob Will...[Keynote] predictive technologies and the prediction of technology - Bob Will...
[Keynote] predictive technologies and the prediction of technology - Bob Will...
PAPIs.io
 
AIAA Conference - Big Data Session_ Final - Jan 2016
AIAA Conference - Big Data Session_ Final - Jan 2016AIAA Conference - Big Data Session_ Final - Jan 2016
AIAA Conference - Big Data Session_ Final - Jan 2016
Manjula Ambur
 
Top Trends Guiding Tech Use in Your Career Practice
Top Trends Guiding Tech Use in Your Career PracticeTop Trends Guiding Tech Use in Your Career Practice
Top Trends Guiding Tech Use in Your Career Practice
Melissa A. Venable
 
Today’s Technology, Tomorrow’s College Student
Today’s Technology, Tomorrow’s College StudentToday’s Technology, Tomorrow’s College Student
Today’s Technology, Tomorrow’s College Student
Leslie Dare
 
Jwcv Jbci 2 P System Tech
Jwcv Jbci 2 P System TechJwcv Jbci 2 P System Tech
Jwcv Jbci 2 P System Tech
jbci
 
Knowledge Engineering, Electronic Government and the applications to Scientom...
Knowledge Engineering, Electronic Government and the applications to Scientom...Knowledge Engineering, Electronic Government and the applications to Scientom...
Knowledge Engineering, Electronic Government and the applications to Scientom...
Roberto C. S. Pacheco
 
Democratizing Science through Cyberinfrastructure - Manish Parashar
Democratizing Science through Cyberinfrastructure - Manish ParasharDemocratizing Science through Cyberinfrastructure - Manish Parashar
Democratizing Science through Cyberinfrastructure - Manish Parashar
Larry Smarr
 
Overview Of Wcu Research (16 Dec2009)Sj
Overview Of Wcu Research (16 Dec2009)SjOverview Of Wcu Research (16 Dec2009)Sj
Overview Of Wcu Research (16 Dec2009)Sj
WCU Webometrics Institute
 
AIIA - Charting the Path to Intelligent Operations with Machine Learning - At...
AIIA - Charting the Path to Intelligent Operations with Machine Learning - At...AIIA - Charting the Path to Intelligent Operations with Machine Learning - At...
AIIA - Charting the Path to Intelligent Operations with Machine Learning - At...
BigML, Inc
 
Insight V25 I4
Insight V25 I4Insight V25 I4
Insight V25 I4
AngelCepedanoBeteta1
 
Final october interviewing_techniques
Final october interviewing_techniquesFinal october interviewing_techniques
Final october interviewing_techniques
Krishnamoorthy Ramakrishnan
 
FAIRification is a Team Sport: FAIRsharing and the FAIR Cookbook
FAIRification is a Team Sport: FAIRsharing and the FAIR CookbookFAIRification is a Team Sport: FAIRsharing and the FAIR Cookbook
FAIRification is a Team Sport: FAIRsharing and the FAIR Cookbook
Susanna-Assunta Sansone
 
IT "The Power That Influence The World"
IT "The Power That Influence The World"IT "The Power That Influence The World"
IT "The Power That Influence The World"
USA Discussion Group
 

Similar to Knowledge engineering: from people to machines and back (20)

Data_Mining_for_Software_Engineering.pdf
Data_Mining_for_Software_Engineering.pdfData_Mining_for_Software_Engineering.pdf
Data_Mining_for_Software_Engineering.pdf
 
Technology Motivators and Usage in Non-Profit Arts Organizations
Technology Motivators and Usage in Non-Profit Arts OrganizationsTechnology Motivators and Usage in Non-Profit Arts Organizations
Technology Motivators and Usage in Non-Profit Arts Organizations
 
siegelCV.doc.doc
siegelCV.doc.docsiegelCV.doc.doc
siegelCV.doc.doc
 
siegelCV.doc.doc
siegelCV.doc.docsiegelCV.doc.doc
siegelCV.doc.doc
 
Bridging Gaps and Broadening Participation in Today's and Future Research Com...
Bridging Gaps and Broadening Participation inToday's and Future Research Com...Bridging Gaps and Broadening Participation inToday's and Future Research Com...
Bridging Gaps and Broadening Participation in Today's and Future Research Com...
 
Chapter 1 Introduction to Datascience (1).pptx
Chapter 1 Introduction to Datascience (1).pptxChapter 1 Introduction to Datascience (1).pptx
Chapter 1 Introduction to Datascience (1).pptx
 
ICSE 2016 - Opening and Awards I
ICSE 2016 - Opening and Awards IICSE 2016 - Opening and Awards I
ICSE 2016 - Opening and Awards I
 
[Keynote] predictive technologies and the prediction of technology - Bob Will...
[Keynote] predictive technologies and the prediction of technology - Bob Will...[Keynote] predictive technologies and the prediction of technology - Bob Will...
[Keynote] predictive technologies and the prediction of technology - Bob Will...
 
AIAA Conference - Big Data Session_ Final - Jan 2016
AIAA Conference - Big Data Session_ Final - Jan 2016AIAA Conference - Big Data Session_ Final - Jan 2016
AIAA Conference - Big Data Session_ Final - Jan 2016
 
Top Trends Guiding Tech Use in Your Career Practice
Top Trends Guiding Tech Use in Your Career PracticeTop Trends Guiding Tech Use in Your Career Practice
Top Trends Guiding Tech Use in Your Career Practice
 
Today’s Technology, Tomorrow’s College Student
Today’s Technology, Tomorrow’s College StudentToday’s Technology, Tomorrow’s College Student
Today’s Technology, Tomorrow’s College Student
 
Jwcv Jbci 2 P System Tech
Jwcv Jbci 2 P System TechJwcv Jbci 2 P System Tech
Jwcv Jbci 2 P System Tech
 
Knowledge Engineering, Electronic Government and the applications to Scientom...
Knowledge Engineering, Electronic Government and the applications to Scientom...Knowledge Engineering, Electronic Government and the applications to Scientom...
Knowledge Engineering, Electronic Government and the applications to Scientom...
 
Democratizing Science through Cyberinfrastructure - Manish Parashar
Democratizing Science through Cyberinfrastructure - Manish ParasharDemocratizing Science through Cyberinfrastructure - Manish Parashar
Democratizing Science through Cyberinfrastructure - Manish Parashar
 
Overview Of Wcu Research (16 Dec2009)Sj
Overview Of Wcu Research (16 Dec2009)SjOverview Of Wcu Research (16 Dec2009)Sj
Overview Of Wcu Research (16 Dec2009)Sj
 
AIIA - Charting the Path to Intelligent Operations with Machine Learning - At...
AIIA - Charting the Path to Intelligent Operations with Machine Learning - At...AIIA - Charting the Path to Intelligent Operations with Machine Learning - At...
AIIA - Charting the Path to Intelligent Operations with Machine Learning - At...
 
Insight V25 I4
Insight V25 I4Insight V25 I4
Insight V25 I4
 
Final october interviewing_techniques
Final october interviewing_techniquesFinal october interviewing_techniques
Final october interviewing_techniques
 
FAIRification is a Team Sport: FAIRsharing and the FAIR Cookbook
FAIRification is a Team Sport: FAIRsharing and the FAIR CookbookFAIRification is a Team Sport: FAIRsharing and the FAIR Cookbook
FAIRification is a Team Sport: FAIRsharing and the FAIR Cookbook
 
IT "The Power That Influence The World"
IT "The Power That Influence The World"IT "The Power That Influence The World"
IT "The Power That Influence The World"
 

More from Elena Simperl

When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
This talk was not generated with ChatGPT: how AI is changing science
This talk was not generated with ChatGPT: how AI is changing scienceThis talk was not generated with ChatGPT: how AI is changing science
This talk was not generated with ChatGPT: how AI is changing science
Elena Simperl
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
Elena Simperl
 
The web of data: how are we doing so far
The web of data: how are we doing so farThe web of data: how are we doing so far
The web of data: how are we doing so far
Elena Simperl
 
What Wikidata teaches us about knowledge engineering
What Wikidata teaches us about knowledge engineeringWhat Wikidata teaches us about knowledge engineering
What Wikidata teaches us about knowledge engineering
Elena Simperl
 
Open government data portals: from publishing to use and impact
Open government data portals: from publishing to use and impactOpen government data portals: from publishing to use and impact
Open government data portals: from publishing to use and impact
Elena Simperl
 
Ten myths about knowledge graphs.pdf
Ten myths about knowledge graphs.pdfTen myths about knowledge graphs.pdf
Ten myths about knowledge graphs.pdf
Elena Simperl
 
What Wikidata teaches us about knowledge engineering
What Wikidata teaches us about knowledge engineeringWhat Wikidata teaches us about knowledge engineering
What Wikidata teaches us about knowledge engineering
Elena Simperl
 
Data commons and their role in fighting misinformation.pdf
Data commons and their role in fighting misinformation.pdfData commons and their role in fighting misinformation.pdf
Data commons and their role in fighting misinformation.pdf
Elena Simperl
 
Are our knowledge graphs trustworthy?
Are our knowledge graphs trustworthy?Are our knowledge graphs trustworthy?
Are our knowledge graphs trustworthy?
Elena Simperl
 
The web of data: how are we doing so far?
The web of data: how are we doing so far?The web of data: how are we doing so far?
The web of data: how are we doing so far?
Elena Simperl
 
Crowdsourcing and citizen engagement for people-centric smart cities
Crowdsourcing and citizen engagement for people-centric smart citiesCrowdsourcing and citizen engagement for people-centric smart cities
Crowdsourcing and citizen engagement for people-centric smart cities
Elena Simperl
 
Pie chart or pizza: identifying chart types and their virality on Twitter
Pie chart or pizza: identifying chart types and their virality on TwitterPie chart or pizza: identifying chart types and their virality on Twitter
Pie chart or pizza: identifying chart types and their virality on Twitter
Elena Simperl
 
High-value datasets: from publication to impact
High-value datasets: from publication to impactHigh-value datasets: from publication to impact
High-value datasets: from publication to impact
Elena Simperl
 
The story of Data Stories
The story of Data StoriesThe story of Data Stories
The story of Data Stories
Elena Simperl
 
The human face of AI: how collective and augmented intelligence can help sol...
The human face of AI:  how collective and augmented intelligence can help sol...The human face of AI:  how collective and augmented intelligence can help sol...
The human face of AI: how collective and augmented intelligence can help sol...
Elena Simperl
 
Qrowd and the city: designing people-centric smart cities
Qrowd and the city: designing people-centric smart citiesQrowd and the city: designing people-centric smart cities
Qrowd and the city: designing people-centric smart cities
Elena Simperl
 
One does not simply crowdsource the Semantic Web: 10 years with people, URIs,...
One does not simply crowdsource the Semantic Web: 10 years with people, URIs,...One does not simply crowdsource the Semantic Web: 10 years with people, URIs,...
One does not simply crowdsource the Semantic Web: 10 years with people, URIs,...
Elena Simperl
 
Qrowd and the city
Qrowd and the cityQrowd and the city
Qrowd and the city
Elena Simperl
 
Inclusive cities: a crowdsourcing approach
Inclusive cities: a crowdsourcing approachInclusive cities: a crowdsourcing approach
Inclusive cities: a crowdsourcing approach
Elena Simperl
 

More from Elena Simperl (20)

When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
This talk was not generated with ChatGPT: how AI is changing science
This talk was not generated with ChatGPT: how AI is changing scienceThis talk was not generated with ChatGPT: how AI is changing science
This talk was not generated with ChatGPT: how AI is changing science
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
The web of data: how are we doing so far
The web of data: how are we doing so farThe web of data: how are we doing so far
The web of data: how are we doing so far
 
What Wikidata teaches us about knowledge engineering
What Wikidata teaches us about knowledge engineeringWhat Wikidata teaches us about knowledge engineering
What Wikidata teaches us about knowledge engineering
 
Open government data portals: from publishing to use and impact
Open government data portals: from publishing to use and impactOpen government data portals: from publishing to use and impact
Open government data portals: from publishing to use and impact
 
Ten myths about knowledge graphs.pdf
Ten myths about knowledge graphs.pdfTen myths about knowledge graphs.pdf
Ten myths about knowledge graphs.pdf
 
What Wikidata teaches us about knowledge engineering
What Wikidata teaches us about knowledge engineeringWhat Wikidata teaches us about knowledge engineering
What Wikidata teaches us about knowledge engineering
 
Data commons and their role in fighting misinformation.pdf
Data commons and their role in fighting misinformation.pdfData commons and their role in fighting misinformation.pdf
Data commons and their role in fighting misinformation.pdf
 
Are our knowledge graphs trustworthy?
Are our knowledge graphs trustworthy?Are our knowledge graphs trustworthy?
Are our knowledge graphs trustworthy?
 
The web of data: how are we doing so far?
The web of data: how are we doing so far?The web of data: how are we doing so far?
The web of data: how are we doing so far?
 
Crowdsourcing and citizen engagement for people-centric smart cities
Crowdsourcing and citizen engagement for people-centric smart citiesCrowdsourcing and citizen engagement for people-centric smart cities
Crowdsourcing and citizen engagement for people-centric smart cities
 
Pie chart or pizza: identifying chart types and their virality on Twitter
Pie chart or pizza: identifying chart types and their virality on TwitterPie chart or pizza: identifying chart types and their virality on Twitter
Pie chart or pizza: identifying chart types and their virality on Twitter
 
High-value datasets: from publication to impact
High-value datasets: from publication to impactHigh-value datasets: from publication to impact
High-value datasets: from publication to impact
 
The story of Data Stories
The story of Data StoriesThe story of Data Stories
The story of Data Stories
 
The human face of AI: how collective and augmented intelligence can help sol...
The human face of AI:  how collective and augmented intelligence can help sol...The human face of AI:  how collective and augmented intelligence can help sol...
The human face of AI: how collective and augmented intelligence can help sol...
 
Qrowd and the city: designing people-centric smart cities
Qrowd and the city: designing people-centric smart citiesQrowd and the city: designing people-centric smart cities
Qrowd and the city: designing people-centric smart cities
 
One does not simply crowdsource the Semantic Web: 10 years with people, URIs,...
One does not simply crowdsource the Semantic Web: 10 years with people, URIs,...One does not simply crowdsource the Semantic Web: 10 years with people, URIs,...
One does not simply crowdsource the Semantic Web: 10 years with people, URIs,...
 
Qrowd and the city
Qrowd and the cityQrowd and the city
Qrowd and the city
 
Inclusive cities: a crowdsourcing approach
Inclusive cities: a crowdsourcing approachInclusive cities: a crowdsourcing approach
Inclusive cities: a crowdsourcing approach
 

Recently uploaded

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
Uni Systems S.M.S.A.
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
mikeeftimakis1
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Zilliz
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
Kari Kakkonen
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Neo4j
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofszkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
Alex Pruden
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Rohit Gautam
 

Recently uploaded (20)

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofszkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
 

Knowledge engineering: from people to machines and back

  • 1. Knowledge engineering: from people to machines and back Elena Simperl, @esimperl
  • 2. Knowledge engineering • Building and maintaining knowledge-based systems • Sits between • Software engineering: how to build and maintain software systems; • AI: how to represent, learn, and reason over knowledge computationally
  • 3. KNOWLEDGE ENGINEERING: BEFORE • Gathering curated knowledge from experts and encoding it into knowledge bases. • Mostly manual process, supported by methodologies, methods, tools. • Not scalable, considerable up- front investment. 3 Source: NeOn project, 2010
  • 4. KNOWLEDGE ENGINEERING: TODAY • Large knowledge bases, drawn from heterogeneous data, automatic process w/ human-in-the-loop. • Mainstream adoption in search, intelligent assistants, digital twins, supply chain management, legal compliance etc. • W/ access to data and (off-the-shelf) AI capabilities, costs are a fraction from what they were decades ago. • New knowledge engineering experiences: prompting, conversational tools, co-auditing automatically generated content 4 Source: Fathallah et al., 2024
  • 9. Two ways forward Automation Involve more people 9
  • 12. People implicitly contribute to the knowledge base while going about their regular work. Benefits are highlighted Challenges: not suitable for every use case, shallow semantics 12
  • 13. People play games. Games data is turned into computational knowledge Challenges: design, attracting and retaining players, costs 13
  • 14. People are part of online communities that build knowledge bases Challenges: uneven participation, data quality 14
  • 15. What if automation were an option? 15 Human computation Large language models
  • 16. Knowledge integrity in Wikidata Wikidata is a widely used secondary source of knowledge Data is trusted when it is backed by its primary source How references are encoded in Wikidata
  • 17. Are references easy to navigate, authoritative, relevant? Yes, but some variations across languages 5/28/2024 17 English, Swedish, Japanese, Spanish, Dutch, Portuguese
  • 18. Does the source support the claim? Yes, but challenging task to automate because of heterogeneity and multimodality of evidence 18
  • 19. (Chandler Fashion Center, directions, Highway 101 & Highway 202) «■ Not logged in Talk Contributions Create account Login voyage Page Discussion Read Edit View history | Search Wikivoyage This is an old revision of this page, as edited by SelfieCity (talk | contribs) at 20:58,13 October 2019 (page banner). (diff) •— Older revision | Latest revision (diff) | Newer revision —» (diff) Main page Travel destinations Star articles What's Nearby? Tourist office Random page North America > United States of America > Chandler Southwest (United States of America) > Arizona > Greater Phoenix > Chandler (Arizona) Travellers' pub Recent changes Community portal Maintenance panel Policies Help Interlingual lounge Donate Chandler is a city in Arizona, and a medium-sized suburb of Phoenix with over 240,000 residents. It is a delightful place to visit. Get in Related changes Upload file By plane Phoenix Sky Harbor International Airport (PHXIATA) +1 602 275-4958 [1]i? is the main air gateway to Arizona. It is in East Phoenix 3 miles from downtown. All major U.S. carriers serve Phoenix Special । Perm am Page inft Cite this F rirt s r * Q Chandler Fashion Center^ (Chandler Mali), 3111 W Chandler Blvd (Highway 101 & Highway 202), W +1 480-812-8488. M-Sa 10AM-9PM, Su 11AM-6PM; restaurant and dept store hours vary An upscale shopping mall with various department stores, (updated Sep 2018) • H me Shoppes at Casa Palomas’, 7131 West Ray Road (Highway 10 & Ray). Upscale outdoor shopping and dining center, (updated Sep 201 B> •BPhoenix Premium Outletss’, 4976 Premium Outlet Way (Highway 10 & Highway 202), +1-480-639-1766. A shopping destination for locals and visitors looking for upscale shopping in a casual, family-friendly atmosphere. (updated Sep 2018) In other projects Get around Wikimedia Commons Wikipedia Car and bus are the easiest ways to get around Chandler Rideshare services Uber and Lytt also operate in the city. Google's Waymo autonomous car service is also available in parts of the city.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26. Datasets WDV: verbalisation WTR: textual references WTR 416 triples, 3 classes (supports, refutes, not-enough-information), free-text rationale to manage diversity of reference layouts
  • 27. Complex task, 87.5% accuracy when explicit textual evidence present 27
  • 29.
  • 34. How do knowledge engineers use LLMs Ethnographic observations of the hackathon Documentary analysis of documents collected during the hackathon Semi-structured interviews with hackathon participants
  • 35. Findings Challenges around lack of datasets and evaluation protocols Guidance and best practices needed on knowledge prompting and auditing LLM outputs Limited awareness of responsible AI concerns like biases, fairness 35
  • 36. LLMs and requirements engineering AI assistants that automate tasks, generate documentation, remove technical jargon 84.1% of knowledge engineers lack access to specialised tools 66.1% lack access to guidelines Manual process, using text editors and spreadsheets [1] Monfardini GK, Salamon JS, Barcellos MP. Use of Competency Questions in Ontology Engineering: A Survey. Int. Conf. on Conceptual Modeling 2023
  • 39. Using LLMs and knowledge graphs to reverse engineer competency questions 39
  • 41. Cards for KGs and KG embeddings
  • 42. Final thoughts • Knowledge engineering is transforming. We need • Methodologies w/ stronger focus on hybrid workflows, new roles (prompting), new skills (co-auditing, responsible AI) • Viable AI copilots, multimodal support, scalable to real knowledge graphs • Datasets, benchmarks, and evaluation protocols to improve the technical performance of LLMs in knowledge engineering tasks (e.g. knowledge prompts, multi-turn dialog, safety, user stories, CQs, documentation etc.) • Campaigns to share and reuse data where possible to reduce data-related costs (prompting collectives) • Guidance, best practices to use AI effectively • Mechanisms to demonstrate AI-co-generated knowledge graphs can be trusted 42
  • 43. Where is the web in all this? 43
  • 44. This talk presented work by: Gabriel Amaral,Jacopo de Berardinis, Fiorela Ciroku, Jonathan Hare, Lucie Kaffee, Jongmo Kim, Elisavet Koutsiana,Albert Meroño-Peñuela, Michelle Nwachukwu,Alessandro Piscopo,Valentina Presutti, Odinaldo Rodrigues, Johanna Walker, Bohui Zhang,Yihang Zhao Thank you also to participants of the Dagstuhl seminar, hackathon, and ESWC special track. Elena Simperl @esimperl 44