The document discusses the construction, growth, and service of large knowledge graphs, emphasizing the role of human-in-the-loop processes to enhance quality and efficiency throughout. It showcases various examples, such as financial content knowledge bases and semantic role labeling, illustrating the application of advanced machine learning techniques and interactive systems for data labeling and extraction. Additionally, it highlights multiple projects and collaborations, including contributions from IBM and academic institutions, aimed at improving knowledge base operations and text analytics.

1. Yunyao Li
HILDA @ SIGMOD’23 | Apple Inc. | June 18, 2023
Building, Growing and Serving
Large Knowledge Graphs with
Human-in-the-Loop

2. Knowledge Bases
Image Source: https://www.csee.umbc.edu/courses/graduate/691/fall22/kg/

3. Example: Financial Content Knowledge Base
Financial
Reports
Ontology
[VLDB’2017] Creation and Interaction with Large-scale Domain-Specific Knowledge Bases.
XML
Knowledge
Extraction
Overall Architecture: A Simplified View
Linking
Fusion
KG Construction
Transforming
>31,000 companies
439 industries
~170,000 insiders
~100 millions financial metrics
~22,000 industry KPIs
Financial Content KB
KG Services
QA
APIs

4. Example: Saga
Structured
Knowledge
Sources
Real-time
Sources
Ontology
Unstructured
Knowledge
Sources
Linking
Fusion
KG Construction KG
Knowledge
Extraction
KG Services
QA
Semantic Annotation
… …
Embedding Services
[SIGMOD’23] Growing and Serving Large Open-domain Knowledge Graphs.
[SIGMOD’22] Saga: A Platform for Continuous Construction and Serving of Knowledge at Scale
Transforming
Overall Architecture: A Simplified View

5. Human-in-the-Loop Throughout the Entire Life Cycle
of KG Construction, Growth, and Services
Data Labeling Development Deployment

6. Human-in-the-Loop Throughout the Entire Life Cycle
of KG Construction, Growth, and Services
Data Labeling Development Deployment
Learner
raw data labeled data
1. Improve quality
2. Increase efficiency
3. Decrease skill requirements

7. Example 1: Scale Fact Collection
Missing / stale facts
Missing
Facts
Query
Synthesizer
QA System
candidate facts
Baseline
New
Facts

8. Example 1: Scale Fact Collection
Missing / stale facts
Missing
Facts
Query
Synthesizer
QA System
candidate facts
Baseline
New
Facts
Query-by-Committee
Missing
Facts
Query
Synthesizer
QA System
candidate facts
New
Facts
QA System
Q1
QA System
… …
… …
…
Qn
QbC
Selector
AnswerSet1
AnswerSetn
[EMNLP-DaSH’2022] Improving Human Annotation Effectiveness for Fact Collection by Identifying the Most Relevant Answers
Success Rate
fact collection
25%

9. Example 1: Scale Fact Collection
Missing / stale facts
Missing
Facts
Query
Synthesizer
QA System
candidate facts
Baseline
New
Facts
Open Domain Knowledge Extraction
[SIGMOD’23] Growing and Serving Large Open-domain Knowledge Graphs.
Throughput vs.
manual fact collection
>100x
Missing
Facts
Query
Synthesizer
Web Search
candidate facts w/
lower-confidence
New
Facts
Knowledge
Extractor
Fact
Corroboration

10. Example 2: Semantic Role Labeling
SRL

11. Example 2a: Crowd-in-the-Loop Curation
An hybrid approach
Corpus
raw data
Corpus
predicated
annotations
Annotation
Task
Corpus
curated
annotations
Task
Router
Difficult tasks are curated by experts
Easier tasks are curated by crowd
[EMNLP’17] CROWD-IN-THE-LOOP: A Hybrid Approach for Annotating Semantic Roles

12. Example 2a: Effectiveness of Crowd-in-the-Loop
9% F1
vs. SRL model
↑ 66.4%
Expert efforts
↓

13. Example 2b: Better Workflow Performs Ever Better
vs. SRL model
↑
Expert efforts
↓
10% F1
vs. SRL model
↑ 87.3%
Expert efforts
↓
Filter
unlikely options
Select
from likely options
Expert
resolve hard cases
[EMNLP’20 (Finding)] A Novel Workflow for Accurately and Efficiently
Crowdsourcing Predicate Senses and Argument Labels

14. Human-in-the-Loop Throughout the Entire Life Cycle
of KG construction, growth, and services
Data Labeling Development Deployment
Learner
Scale data labeling
raw data labeled data
IDE
Better IDE for model building

15. Different Tooling for Different Users
Full-fledged IDE
AI Engineers AI Engineers/Data Scientists
Visual IDE
[ACL’12] WizIE: A Best Practices Guided Development Environment for
Information Extraction
[CHI’13] I can do text analytics!: designing development tools for novice
developers
[VLDB’15] VINERy: A Visual IDE for Information Extraction
[KDD’19] Declarative Text Understanding with SystemT. (hands-on tutorial)
Entity Extraction in AIOps https://www.ibm.com/cloud/blog/entity-extraction-in-aiops
IBM InfoSphere BigInsights Text Analytics Eclipse Tooling
IBM Watson Knowledge Studio. Advanced Rule Editor http://ibm.biz/VineryIE

16. Human-in-the-Loop Throughout the Entire Life Cycle
of KG construction, growth, and services
Data Labeling Development Deployment
Learner
Scale data labeling
raw data labeled data
IDE
Better IDE for model building
Learner
Human-machine co-creation

17. Transparent Linguistic Models for Contract Understanding
Watson Discovery Content Intelligence
[NAACL’21] Development of an Enterprise-Grade Contract Understanding System, (Industry Track)

18. HEIDL: Human & Machine Co-Creation via Neural-Symbolic AI
[ACL’19] HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop.
[EMNLP’20] Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification
In use for major IBM customer engagements
Raises the abstraction level for domain experts to interact with

19. HEIDL: Human & Machine Co-Creation via Neural-Symbolic AI
[ACL’19] HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop.
[EMNLP’20] Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification
In use for major IBM customer engagements
Raises the abstraction level for domain experts to interact with

20. Human-in-the-Loop Throughout the Entire Life Cycle
of KG construction, growth, and services
Data Labeling Development Deployment
Learner
Scale data labeling
raw data labeled data
IDE
Better IDE for model building
Learner
Human-machine co-creation
Learner
Curb data hunger with interactive learning

21. Case 1: Example-Driven Extraction
Via pattern induction
[CHI’17] SEER: Auto-Generating Information Extraction Rules from User-Specified Examples
[SIGMOD’17] Synthesizing Extraction Rules from User Examples with SEER. SIGMOD’2017
[AAAI’22 (demo)] InteractEva: A Simulation-based Evaluation Framework for Interactive AI Systems
[AAAI’22] A Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application.
IBM Watson Discovery (Beta in Plus since Oct. 2021) http://ibm.biz/SEER_IE, https://ibm.biz/WDSPressReleaseNov

22. Case 1: Example-Driven Extraction
Via pattern induction
[CHI’17] SEER: Auto-Generating Information Extraction Rules from User-Specified Examples
[SIGMOD’17] Synthesizing Extraction Rules from User Examples with SEER. SIGMOD’2017
[AAAI’22 (demo)] InteractEva: A Simulation-based Evaluation Framework for Interactive AI Systems
[AAAI’22] A Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application.
IBM Watson Discovery (Beta in Plus since Oct. 2021) http://ibm.biz/SEER_IE, https://ibm.biz/WDSPressReleaseNov

23. Case 1: Example-Driven Extraction
Via pattern induction
[CHI’17] SEER: Auto-Generating Information Extraction Rules from User-Specified Examples
[SIGMOD’17] Synthesizing Extraction Rules from User Examples with SEER. SIGMOD’2017
[AAAI’22 (demo)] InteractEva: A Simulation-based Evaluation Framework for Interactive AI Systems
[AAAI’22] A Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application.
IBM Watson Discovery (Beta in Plus since Oct. 2021) http://ibm.biz/SEER_IE, https://ibm.biz/WDSPressReleaseNov

24. Case 1: Example-Driven Extraction
Via pattern induction
[CHI’17] SEER: Auto-Generating Information Extraction Rules from User-Specified Examples
[SIGMOD’17] Synthesizing Extraction Rules from User Examples with SEER. SIGMOD’2017
[AAAI’22 (demo)] InteractEva: A Simulation-based Evaluation Framework for Interactive AI Systems
[AAAI’22] A Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application.
IBM Watson Discovery (Beta in Plus since Oct. 2021) http://ibm.biz/SEER_IE, https://ibm.biz/WDSPressReleaseNov

25. Case 2: Entity Normalization & Variant Generation
Learning Structured Representations
Capture Entity Semantic Structure
[COLING’2018] Exploiting Structure in Representation of Named Entities using Active Learning.
[ICDE’2018] LUSTRE: An Interactive System for Entity Structured Representation and Variant
Generation.
Generated normalizers for Watson Discovery
[AAAI’2020] PARTNER: Human-in-the-Loop Entity Name Understanding with Deep
Learning.
[EMNLP’2020] Learning Structured Representations of Entity Names using Active
Learning and Weak Supervision.
“Bank of America N.A.” “Bank of America National Association”
Synthesizing Normalization and
Variant Generation Functions

26. Case 3: Deep Document Understanding
Document Ingestion
[WACV 2021] Global Table Extractor (GTE): A Framework for Joint Table Identification
and Cell Structure Recognition Using Visual Context.
[AAAI’21] KAAPA: Knowledge Aware Answers from PDF Analysis.
[ACL-CORD-19’21] CORD-19: The COVID-19 Open Research Dataset
Bringing IBM NLP capabilities to the CORD-19 Dataset. http://ibm.biz/CORD19-IBM
IBM Watson Discovery
JSON/HTML
Wide Variety in PDF Tables
Table with
graphic lines
Table with
visual clues only
Complex
table with
multi-row/column
headers
Table interleaved
with text and charts

27. Case 3: TableLab
TableLab: Easy Customization via Adaptive Deep Learning
[IUI’2021] TableLab: An Interactive Table Extraction System with Adaptive Deep Learning.

28. Case 3: Deep Document Understanding
TableLab: Easy Customization via Adaptive Deep Learning
[IUI’2021] TableLab: An Interactive Table Extraction System with Adaptive Deep Learning.

29. Case 3: Customization vis TableLab
Table Boundary Detection
Preliminary Results
Method CEDAR EDGAR Invoices Appraisals Health
Docs
GTE 0.94 0.84 0.47 0.85 0.93
GTE with
Retraining
0.96 0.91 0.92 0.96 0.98
Method CEDAR EDGAR Invoices Appraisals Health
Docs
GTE 0.88 0.62 0.42 0.71 0.55
GTE with
Retraining
0.90 0.82 0.68 0.90 0.77
Cell Adjacency Detection
Dataset
20 pages with tables per category: 10 for
retraining, 10 for testing
Evaluation Metric
F1 metric for Table Boundary and Cell Adjacency
as de
fi
ned in [1]
[1] Göbel et al. “A Methodology for Evaluating Algorithms for Table Understanding in PDF
Documents”. DocEng '12

30. Case 4: Label Sleuth
An open-source no-code system for text annotation and building text classifiers
[EMNLP’2022] Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours
https://www.label-sleuth.org
1. From task definition to working
model in hours!
2. Extensible backend to integrate new
model architectures or active
learning techniques

31. Human-in-the-Loop Throughout the Entire Life Cycle
of KG construction, growth, and services
Data Labeling Development Deployment
Learner
Scale data labeling
raw data labeled data
IDE
Better IDE for model building
Learner
Human-machine co-creation
Learner
Curb data hunger with interactive learning
AutoML
Scale model building via AutoML

32. AutoAI for Text
AutoText
[AAAI’21] AutoText: An End-to-End AutoAI Framework for Text.
[NeurIPS 2022] AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning.
IBM Developer API https://developer.ibm.com/learningpaths/get-started-autoai-for-text-api
Example Use Case: Scale ML Product
Model for Text Classification
>30%
Reduction in combined
training and prediction
time
Auto weight
tuning & HPO
>10x
Speed-up in training at
comparable quality
Auto classifier
selection

33. Human-in-the-Loop Throughout the Entire Life Cycle
of KG construction, growth, and services
Data Labeling Development Deployment
Learner
Scale data labeling
raw data labeled data
IDE
Better IDE for model building
Learner
Human-machine co-creation
Learner
Curb data hunger with interactive learning
AutoML
Scale model building via AutoML

34. Human-in-the-Loop Throughout the Entire Life Cycle
of KG construction, growth, and services
Data Labeling Development Deployment
Learner
Scale data labeling
raw data labeled data
IDE
Better IDE for model building
Learner
Human-machine co-creation
Learner
Curb data hunger with interactive learning
AutoML
Scale model building via AutoML
Query
Log Tickets
User feedback influence the entire life cycle

35. Quality Evaluation
1. Measure what matters for end users
2. Identify the root cause of failures
3. Track improvements in individual
components as they evolve
The key requirements
- Who won the Paris
Paris, France Paris Masters

36. Overall Evaluation Framework
A Human-in-the-Loop Process
Annotation Quality Metrics
Dataset Collection
Tooling and annotation guidelines
for graders
Evaluation
Human in the loop to annotate/
grade queries
Logs
Synthetic Queries
Knowledge Graph Metrics
End to End Metrics
Query Understanding Metrics

37. Visual Tooling of Metrics and Loss Buckets
- Example Errors:
- Entity Prediction Error: “Who won Paris” (Paris Masters/Paris–Roubaix)
- Missing Fact: “When is the oscars in 2026”
- Fact is not present because date/location is not published yet)
- Unrecognized Entity in KG: ”Who is princess noor horse”
Facilitate Opportunity Analysis

38. Visual Diagnosis
[DaSH@KDD. 2020] WhyFlow: Explaining Errors in Data Flows Interactively.

39. ModelLens
Visual interactive tool for model improvement
[CSCW’19] ModelLens: An Interactive System to Support the Model Improvement Practices of Data Science Teams.

40. So how will EVERYTHING
change with LLMs?
Many exciting challenges and opportunities

41. Thanks!
IBM (including interns):
• Shivakumar Vaithyanathan
• Lucian Popa
• Ron Fagin
• Sriram Raghavan
• Rajasekar Krishnamurthy
• Fred Reiss
• Laura Chiticariu
• Benny Kimelfeld
• Mauricio Hernadez
• Eser Kandogan
• Huaiyu Zhu
• Kun Qian
• Dakuo Wang
• Maeda Hanafi
Many amazing collaborators and interns …
Apple (including interns):
• Ihab Ilyas
• Theodoros Rekatsinas
• Umar Farooq Minhas
• Ali Mousavi
• Jefferey Pound
• Anil Pacaci
• Shihabur R. Chowdhury
• Hongyu Ren
• Jason Mohoney
• Kun Qian
• Yiwen Sun
• Yisi Sang
• Saloni Potdar
• … …
Universities:
• Azza Abouzeid (NYU-Abu Dhabi)
• H. V. Jagadish (U. Of Michigan)
• Fei Xia (U. Of Washington)
• Kevin Chen-Chuan Chang (UIUC)
• ChengXiang Zhai (UIUC)
• Domenico Lembo(Sapienza
University of Rome)
• Dragomir R. Radev (Yale)
• Jonathan K. Kummerfeld (U. Of
Michigan)
• Walter S. Lasecki (U. Of Michigan)
• Toby Li (U. of Notre Dame)
• Rishabh Iyer (UT Dallas)
• Eduard C. Dragut (Temple Univ.)
• … ….
• Douglas Burdick’
• Alan Akbik
• Nancy Wang
• Prithiviraj Sen
• Marina Danilevsky
• Poornima Chozhiyath Raman
• Sudarshan Rangarajan
• Ramiya Venkatachalam
• Kiran Kate
• Eyal Shnarch
• Ishan Jindal
• Yiwei Yang
• Nikita Bhutani
• … ….