SlideShare a Scribd company logo

Building, Growing and Serving Large Knowledge Graphs with Human-in-the-Loop

•
•
Yunyao Li
Yunyao Li

Keynote talk at HILDA'2023 at SIGMOD on June 18, 2023. Abstract: The ability to build large-scale knowledge bases that capture and extend the implicit knowledge of human experts is the foundation for many AI systems. We use an ontology-driven approach for the building, growing and serving of such knowledge bases. This approach relies on several well-known building blocks: document conversion, natural language processing, entity resolution, data transformation and fusion. In this talk, I will discuss wide range of real-world challenges related to the building of these blocks and present our work to address these challenges via better human-machine cooperation.

1 of 41
Download to read offline
Yunyao Li HILDA @ SIGMOD’23 | Apple Inc. | June 18, 2023 Building, Growing and Serving Large Knowledge Graphs with Human-in-the-Loop
Knowledge Bases Image Source: https://www.csee.umbc.edu/courses/graduate/691/fall22/kg/
Example: Financial Content Knowledge Base Financial Reports Ontology [VLDB’2017] Creation and Interaction with Large-scale Domain-Specific Knowledge Bases. XML Knowledge Extraction Overall Architecture: A Simplified View Linking Fusion KG Construction Transforming >31,000 companies 439 industries ~170,000 insiders ~100 millions financial metrics ~22,000 industry KPIs Financial Content KB KG Services QA APIs
Example: Saga Structured Knowledge Sources Real-time Sources Ontology Unstructured Knowledge Sources Linking Fusion KG Construction KG Knowledge Extraction KG Services QA Semantic Annotation … … Embedding Services [SIGMOD’23] Growing and Serving Large Open-domain Knowledge Graphs. [SIGMOD’22] Saga: A Platform for Continuous Construction and Serving of Knowledge at Scale Transforming Overall Architecture: A Simplified View
Human-in-the-Loop Throughout the Entire Life Cycle of KG Construction, Growth, and Services Data Labeling Development Deployment
Human-in-the-Loop Throughout the Entire Life Cycle of KG Construction, Growth, and Services Data Labeling Development Deployment Learner raw data labeled data 1. Improve quality 2. Increase efficiency 3. Decrease skill requirements
Example 1: Scale Fact Collection Missing / stale facts Missing Facts Query Synthesizer QA System candidate facts Baseline New Facts
Example 1: Scale Fact Collection Missing / stale facts Missing Facts Query Synthesizer QA System candidate facts Baseline New Facts Query-by-Committee Missing Facts Query Synthesizer QA System candidate facts New Facts QA System Q1 QA System … … … … … Qn QbC Selector AnswerSet1 AnswerSetn [EMNLP-DaSH’2022] Improving Human Annotation Effectiveness for Fact Collection by Identifying the Most Relevant Answers Success Rate fact collection 25%
Example 1: Scale Fact Collection Missing / stale facts Missing Facts Query Synthesizer QA System candidate facts Baseline New Facts Open Domain Knowledge Extraction [SIGMOD’23] Growing and Serving Large Open-domain Knowledge Graphs. Throughput vs. manual fact collection >100x Missing Facts Query Synthesizer Web Search candidate facts w/ lower-confidence New Facts Knowledge Extractor Fact Corroboration
Example 2: Semantic Role Labeling SRL
Example 2a: Crowd-in-the-Loop Curation An hybrid approach Corpus raw data Corpus predicated annotations Annotation Task Corpus curated annotations Task Router Difficult tasks are curated by experts Easier tasks are curated by crowd [EMNLP’17] CROWD-IN-THE-LOOP: A Hybrid Approach for Annotating Semantic Roles
Example 2a: Effectiveness of Crowd-in-the-Loop 9% F1 vs. SRL model ↑ 66.4% Expert efforts ↓
Example 2b: Better Workflow Performs Ever Better vs. SRL model ↑ Expert efforts ↓ 10% F1 vs. SRL model ↑ 87.3% Expert efforts ↓ Filter unlikely options Select from likely options Expert resolve hard cases [EMNLP’20 (Finding)] A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels
Human-in-the-Loop Throughout the Entire Life Cycle of KG construction, growth, and services Data Labeling Development Deployment Learner Scale data labeling raw data labeled data IDE Better IDE for model building
Different Tooling for Different Users Full-fledged IDE AI Engineers AI Engineers/Data Scientists Visual IDE [ACL’12] WizIE: A Best Practices Guided Development Environment for Information Extraction [CHI’13] I can do text analytics!: designing development tools for novice developers [VLDB’15] VINERy: A Visual IDE for Information Extraction [KDD’19] Declarative Text Understanding with SystemT. (hands-on tutorial) Entity Extraction in AIOps https://www.ibm.com/cloud/blog/entity-extraction-in-aiops IBM InfoSphere BigInsights Text Analytics Eclipse Tooling IBM Watson Knowledge Studio. Advanced Rule Editor http://ibm.biz/VineryIE
Human-in-the-Loop Throughout the Entire Life Cycle of KG construction, growth, and services Data Labeling Development Deployment Learner Scale data labeling raw data labeled data IDE Better IDE for model building Learner Human-machine co-creation
Transparent Linguistic Models for Contract Understanding Watson Discovery Content Intelligence [NAACL’21] Development of an Enterprise-Grade Contract Understanding System, (Industry Track)
HEIDL: Human & Machine Co-Creation via Neural-Symbolic AI [ACL’19] HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop. [EMNLP’20] Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification In use for major IBM customer engagements Raises the abstraction level for domain experts to interact with
HEIDL: Human & Machine Co-Creation via Neural-Symbolic AI [ACL’19] HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop. [EMNLP’20] Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification In use for major IBM customer engagements Raises the abstraction level for domain experts to interact with
Human-in-the-Loop Throughout the Entire Life Cycle of KG construction, growth, and services Data Labeling Development Deployment Learner Scale data labeling raw data labeled data IDE Better IDE for model building Learner Human-machine co-creation Learner Curb data hunger with interactive learning
Case 1: Example-Driven Extraction Via pattern induction [CHI’17] SEER: Auto-Generating Information Extraction Rules from User-Specified Examples [SIGMOD’17] Synthesizing Extraction Rules from User Examples with SEER. SIGMOD’2017 [AAAI’22 (demo)] InteractEva: A Simulation-based Evaluation Framework for Interactive AI Systems [AAAI’22] A Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application. IBM Watson Discovery (Beta in Plus since Oct. 2021) http://ibm.biz/SEER_IE, https://ibm.biz/WDSPressReleaseNov
Case 1: Example-Driven Extraction Via pattern induction [CHI’17] SEER: Auto-Generating Information Extraction Rules from User-Specified Examples [SIGMOD’17] Synthesizing Extraction Rules from User Examples with SEER. SIGMOD’2017 [AAAI’22 (demo)] InteractEva: A Simulation-based Evaluation Framework for Interactive AI Systems [AAAI’22] A Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application. IBM Watson Discovery (Beta in Plus since Oct. 2021) http://ibm.biz/SEER_IE, https://ibm.biz/WDSPressReleaseNov
Case 1: Example-Driven Extraction Via pattern induction [CHI’17] SEER: Auto-Generating Information Extraction Rules from User-Specified Examples [SIGMOD’17] Synthesizing Extraction Rules from User Examples with SEER. SIGMOD’2017 [AAAI’22 (demo)] InteractEva: A Simulation-based Evaluation Framework for Interactive AI Systems [AAAI’22] A Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application. IBM Watson Discovery (Beta in Plus since Oct. 2021) http://ibm.biz/SEER_IE, https://ibm.biz/WDSPressReleaseNov
Case 1: Example-Driven Extraction Via pattern induction [CHI’17] SEER: Auto-Generating Information Extraction Rules from User-Specified Examples [SIGMOD’17] Synthesizing Extraction Rules from User Examples with SEER. SIGMOD’2017 [AAAI’22 (demo)] InteractEva: A Simulation-based Evaluation Framework for Interactive AI Systems [AAAI’22] A Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application. IBM Watson Discovery (Beta in Plus since Oct. 2021) http://ibm.biz/SEER_IE, https://ibm.biz/WDSPressReleaseNov
Case 2: Entity Normalization & Variant Generation Learning Structured Representations Capture Entity Semantic Structure [COLING’2018] Exploiting Structure in Representation of Named Entities using Active Learning. [ICDE’2018] LUSTRE: An Interactive System for Entity Structured Representation and Variant Generation. Generated normalizers for Watson Discovery [AAAI’2020] PARTNER: Human-in-the-Loop Entity Name Understanding with Deep Learning. [EMNLP’2020] Learning Structured Representations of Entity Names using Active Learning and Weak Supervision. “Bank of America N.A.” “Bank of America National Association” Synthesizing Normalization and Variant Generation Functions
Case 3: Deep Document Understanding Document Ingestion [WACV 2021] Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context. [AAAI’21] KAAPA: Knowledge Aware Answers from PDF Analysis. [ACL-CORD-19’21] CORD-19: The COVID-19 Open Research Dataset Bringing IBM NLP capabilities to the CORD-19 Dataset. http://ibm.biz/CORD19-IBM IBM Watson Discovery JSON/HTML Wide Variety in PDF Tables Table with graphic lines Table with visual clues only Complex table with multi-row/column headers Table interleaved with text and charts
Case 3: TableLab TableLab: Easy Customization via Adaptive Deep Learning [IUI’2021] TableLab: An Interactive Table Extraction System with Adaptive Deep Learning.
Case 3: Deep Document Understanding TableLab: Easy Customization via Adaptive Deep Learning [IUI’2021] TableLab: An Interactive Table Extraction System with Adaptive Deep Learning.
Case 3: Customization vis TableLab Table Boundary Detection Preliminary Results Method CEDAR EDGAR Invoices Appraisals Health Docs GTE 0.94 0.84 0.47 0.85 0.93 GTE with Retraining 0.96 0.91 0.92 0.96 0.98 Method CEDAR EDGAR Invoices Appraisals Health Docs GTE 0.88 0.62 0.42 0.71 0.55 GTE with Retraining 0.90 0.82 0.68 0.90 0.77 Cell Adjacency Detection Dataset 20 pages with tables per category: 10 for retraining, 10 for testing Evaluation Metric F1 metric for Table Boundary and Cell Adjacency as de fi ned in [1] [1] Göbel et al. “A Methodology for Evaluating Algorithms for Table Understanding in PDF Documents”. DocEng '12
Case 4: Label Sleuth An open-source no-code system for text annotation and building text classifiers [EMNLP’2022] Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours https://www.label-sleuth.org 1. From task definition to working model in hours! 2. Extensible backend to integrate new model architectures or active learning techniques
Human-in-the-Loop Throughout the Entire Life Cycle of KG construction, growth, and services Data Labeling Development Deployment Learner Scale data labeling raw data labeled data IDE Better IDE for model building Learner Human-machine co-creation Learner Curb data hunger with interactive learning AutoML Scale model building via AutoML
AutoAI for Text AutoText [AAAI’21] AutoText: An End-to-End AutoAI Framework for Text. [NeurIPS 2022] AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning. IBM Developer API https://developer.ibm.com/learningpaths/get-started-autoai-for-text-api Example Use Case: Scale ML Product Model for Text Classification >30% Reduction in combined training and prediction time Auto weight tuning & HPO >10x Speed-up in training at comparable quality Auto classifier selection
Human-in-the-Loop Throughout the Entire Life Cycle of KG construction, growth, and services Data Labeling Development Deployment Learner Scale data labeling raw data labeled data IDE Better IDE for model building Learner Human-machine co-creation Learner Curb data hunger with interactive learning AutoML Scale model building via AutoML
Human-in-the-Loop Throughout the Entire Life Cycle of KG construction, growth, and services Data Labeling Development Deployment Learner Scale data labeling raw data labeled data IDE Better IDE for model building Learner Human-machine co-creation Learner Curb data hunger with interactive learning AutoML Scale model building via AutoML Query Log Tickets User feedback influence the entire life cycle
Quality Evaluation 1. Measure what matters for end users 2. Identify the root cause of failures 3. Track improvements in individual components as they evolve The key requirements - Who won the Paris Paris, France Paris Masters
Overall Evaluation Framework A Human-in-the-Loop Process Annotation Quality Metrics Dataset Collection Tooling and annotation guidelines for graders Evaluation Human in the loop to annotate/ grade queries Logs Synthetic Queries Knowledge Graph Metrics End to End Metrics Query Understanding Metrics
Visual Tooling of Metrics and Loss Buckets - Example Errors: - Entity Prediction Error: “Who won Paris” (Paris Masters/Paris–Roubaix) - Missing Fact: “When is the oscars in 2026” - Fact is not present because date/location is not published yet) - Unrecognized Entity in KG: ”Who is princess noor horse” Facilitate Opportunity Analysis
Visual Diagnosis [DaSH@KDD. 2020] WhyFlow: Explaining Errors in Data Flows Interactively.
ModelLens Visual interactive tool for model improvement [CSCW’19] ModelLens: An Interactive System to Support the Model Improvement Practices of Data Science Teams.
So how will EVERYTHING change with LLMs? Many exciting challenges and opportunities
Thanks! IBM (including interns): • Shivakumar Vaithyanathan • Lucian Popa • Ron Fagin • Sriram Raghavan • Rajasekar Krishnamurthy • Fred Reiss • Laura Chiticariu • Benny Kimelfeld • Mauricio Hernadez • Eser Kandogan • Huaiyu Zhu • Kun Qian • Dakuo Wang • Maeda Hanafi Many amazing collaborators and interns … Apple (including interns): • Ihab Ilyas • Theodoros Rekatsinas • Umar Farooq Minhas • Ali Mousavi • Jefferey Pound • Anil Pacaci • Shihabur R. Chowdhury • Hongyu Ren • Jason Mohoney • Kun Qian • Yiwen Sun • Yisi Sang • Saloni Potdar • … … Universities: • Azza Abouzeid (NYU-Abu Dhabi) • H. V. Jagadish (U. Of Michigan) • Fei Xia (U. Of Washington) • Kevin Chen-Chuan Chang (UIUC) • ChengXiang Zhai (UIUC) • Domenico Lembo(Sapienza University of Rome) • Dragomir R. Radev (Yale) • Jonathan K. Kummerfeld (U. Of Michigan) • Walter S. Lasecki (U. Of Michigan) • Toby Li (U. of Notre Dame) • Rishabh Iyer (UT Dallas) • Eduard C. Dragut (Temple Univ.) • … …. • Douglas Burdick’ • Alan Akbik • Nancy Wang • Prithiviraj Sen • Marina Danilevsky • Poornima Chozhiyath Raman • Sudarshan Rangarajan • Ramiya Venkatachalam • Kiran Kate • Eyal Shnarch • Ishan Jindal • Yiwei Yang • Nikita Bhutani • … ….

Recommended

2. BigQuery ML を用いた時系列データの解析 (ARIMA model)
2. BigQuery ML を用いた時系列データの解析 (ARIMA model)2. BigQuery ML を用いた時系列データの解析 (ARIMA model)
2. BigQuery ML を用いた時系列データの解析 (ARIMA model)
幸太朗 岩澤
 
Solving Quantitative Reasoning Problems with Language Models
Solving Quantitative Reasoning Problems with Language ModelsSolving Quantitative Reasoning Problems with Language Models
Solving Quantitative Reasoning Problems with Language Models
harmonylab
 
PythonとRによるデータ分析環境の構築と機械学習によるデータ認識　第3版
PythonとRによるデータ分析環境の構築と機械学習によるデータ認識　第3版PythonとRによるデータ分析環境の構築と機械学習によるデータ認識　第3版
PythonとRによるデータ分析環境の構築と機械学習によるデータ認識　第3版
Katsuhiro Morishita
 
Learning to summarize from human feedback
Learning to summarize from human feedbackLearning to summarize from human feedback
Learning to summarize from human feedback
harmonylab
 
機械学習モデルの判断根拠の説明
機械学習モデルの判断根拠の説明機械学習モデルの判断根拠の説明
機械学習モデルの判断根拠の説明
Satoshi Hara
 
【DL輪読会】GAN-Supervised Dense Visual Alignment (CVPR 2022)
【DL輪読会】GAN-Supervised Dense Visual Alignment (CVPR 2022)【DL輪読会】GAN-Supervised Dense Visual Alignment (CVPR 2022)
【DL輪読会】GAN-Supervised Dense Visual Alignment (CVPR 2022)
Deep Learning JP
 
Graph R-CNN for Scene Graph Generation
Graph R-CNN for Scene Graph GenerationGraph R-CNN for Scene Graph Generation
Graph R-CNN for Scene Graph Generation
Sangmin Woo
 
BlackBox モデルの説明性・解釈性技術の実装
BlackBox モデルの説明性・解釈性技術の実装BlackBox モデルの説明性・解釈性技術の実装
BlackBox モデルの説明性・解釈性技術の実装
Deep Learning Lab（ディープラーニング・ラボ）
 

Recommended

2. BigQuery ML を用いた時系列データの解析 (ARIMA model)
2. BigQuery ML を用いた時系列データの解析 (ARIMA model)2. BigQuery ML を用いた時系列データの解析 (ARIMA model)
2. BigQuery ML を用いた時系列データの解析 (ARIMA model)
幸太朗 岩澤
 
Solving Quantitative Reasoning Problems with Language Models
Solving Quantitative Reasoning Problems with Language ModelsSolving Quantitative Reasoning Problems with Language Models
Solving Quantitative Reasoning Problems with Language Models
harmonylab
 
PythonとRによるデータ分析環境の構築と機械学習によるデータ認識　第3版
PythonとRによるデータ分析環境の構築と機械学習によるデータ認識　第3版PythonとRによるデータ分析環境の構築と機械学習によるデータ認識　第3版
PythonとRによるデータ分析環境の構築と機械学習によるデータ認識　第3版
Katsuhiro Morishita
 
Learning to summarize from human feedback
Learning to summarize from human feedbackLearning to summarize from human feedback
Learning to summarize from human feedback
harmonylab
 
機械学習モデルの判断根拠の説明
機械学習モデルの判断根拠の説明機械学習モデルの判断根拠の説明
機械学習モデルの判断根拠の説明
Satoshi Hara
 
【DL輪読会】GAN-Supervised Dense Visual Alignment (CVPR 2022)
【DL輪読会】GAN-Supervised Dense Visual Alignment (CVPR 2022)【DL輪読会】GAN-Supervised Dense Visual Alignment (CVPR 2022)
【DL輪読会】GAN-Supervised Dense Visual Alignment (CVPR 2022)
Deep Learning JP
 
Graph R-CNN for Scene Graph Generation
Graph R-CNN for Scene Graph GenerationGraph R-CNN for Scene Graph Generation
Graph R-CNN for Scene Graph Generation
Sangmin Woo
 
BlackBox モデルの説明性・解釈性技術の実装
BlackBox モデルの説明性・解釈性技術の実装BlackBox モデルの説明性・解釈性技術の実装
BlackBox モデルの説明性・解釈性技術の実装
Deep Learning Lab（ディープラーニング・ラボ）
 
情報システム障害解析のための知識グラフ構築の試み / Constructing a knowledge graph for information sys...
情報システム障害解析のための知識グラフ構築の試み / Constructing a knowledge graph for information sys...情報システム障害解析のための知識グラフ構築の試み / Constructing a knowledge graph for information sys...
情報システム障害解析のための知識グラフ構築の試み / Constructing a knowledge graph for information sys...
Shinji Takao
 
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "
Deep Learning JP
 
NVIDIA Seminar ディープラーニングによる画像認識と応用事例
NVIDIA Seminar ディープラーニングによる画像認識と応用事例NVIDIA Seminar ディープラーニングによる画像認識と応用事例
NVIDIA Seminar ディープラーニングによる画像認識と応用事例
Takayoshi Yamashita
 
BERTによる文書系AIの取り組みと、Azureを用いたテーブルデータの説明性実現！
BERTによる文書系AIの取り組みと、Azureを用いたテーブルデータの説明性実現！BERTによる文書系AIの取り組みと、Azureを用いたテーブルデータの説明性実現！
BERTによる文書系AIの取り組みと、Azureを用いたテーブルデータの説明性実現！
Deep Learning Lab（ディープラーニング・ラボ）
 
翻訳精度の最大化による同時音声翻訳のための文分割法 (NLP2014)
翻訳精度の最大化による同時音声翻訳のための文分割法 (NLP2014)翻訳精度の最大化による同時音声翻訳のための文分割法 (NLP2014)
翻訳精度の最大化による同時音声翻訳のための文分割法 (NLP2014)
Yusuke Oda
 
ダウンヒルシンプレックス法について解説
ダウンヒルシンプレックス法について解説ダウンヒルシンプレックス法について解説
ダウンヒルシンプレックス法について解説
Kenyu Uehara
 
[DL輪読会]EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
[DL輪読会]EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning[DL輪読会]EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
[DL輪読会]EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
Deep Learning JP
 
AI勉強会用スライド
AI勉強会用スライドAI勉強会用スライド
AI勉強会用スライド
harmonylab
 
CycleGANで顔写真をアニメ調に変換する
CycleGANで顔写真をアニメ調に変換するCycleGANで顔写真をアニメ調に変換する
CycleGANで顔写真をアニメ調に変換する
meownoisy
 
[DL輪読会]Understanding Black-box Predictions via Influence Functions
[DL輪読会]Understanding Black-box Predictions via Influence Functions [DL輪読会]Understanding Black-box Predictions via Influence Functions
[DL輪読会]Understanding Black-box Predictions via Influence Functions
Deep Learning JP
 
【DL輪読会】StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
【DL輪読会】StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery【DL輪読会】StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
【DL輪読会】StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Deep Learning JP
 
[DL輪読会]Causality Inspired Representation Learning for Domain Generalization
[DL輪読会]Causality Inspired Representation Learning for Domain Generalization[DL輪読会]Causality Inspired Representation Learning for Domain Generalization
[DL輪読会]Causality Inspired Representation Learning for Domain Generalization
Deep Learning JP
 
Link prediction with the linkpred tool
Link prediction with the linkpred toolLink prediction with the linkpred tool
Link prediction with the linkpred tool
Raf Guns
 
[DL Hacks]CelebAをNumPyで保存してみた
[DL Hacks]CelebAをNumPyで保存してみた[DL Hacks]CelebAをNumPyで保存してみた
[DL Hacks]CelebAをNumPyで保存してみた
Deep Learning JP
 
[DLHacks]StyleGANとBigGANのStyle mixing, morphing
[DLHacks]StyleGANとBigGANのStyle mixing, morphing[DLHacks]StyleGANとBigGANのStyle mixing, morphing
[DLHacks]StyleGANとBigGANのStyle mixing, morphing
Deep Learning JP
 
[DL輪読会] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
[DL輪読会] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields [DL輪読会] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
[DL輪読会] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
Deep Learning JP
 
研究発表のやり方
研究発表のやり方研究発表のやり方
研究発表のやり方
Akinori Ito
 
【DL輪読会】Scaling Laws for Neural Language Models
【DL輪読会】Scaling Laws for Neural Language Models【DL輪読会】Scaling Laws for Neural Language Models
【DL輪読会】Scaling Laws for Neural Language Models
Deep Learning JP
 
論文紹介「A Perspective View and Survey of Meta-Learning」
論文紹介「A Perspective View and Survey of Meta-Learning」論文紹介「A Perspective View and Survey of Meta-Learning」
論文紹介「A Perspective View and Survey of Meta-Learning」
Kota Matsui
 
【DL輪読会】"A Generalist Agent"
【DL輪読会】"A Generalist Agent"【DL輪読会】"A Generalist Agent"
【DL輪読会】"A Generalist Agent"
Deep Learning JP
 
Information Technology in Industry(ITII) - November Issue 2018
Information Technology in Industry(ITII) - November Issue 2018Information Technology in Industry(ITII) - November Issue 2018
Information Technology in Industry(ITII) - November Issue 2018
ITIIIndustries
 
Digital Twin and Smart Spaces
Digital Twin and Smart Spaces Digital Twin and Smart Spaces
Digital Twin and Smart Spaces
SANGHEE SHIN
 

More Related Content

What's hot

情報システム障害解析のための知識グラフ構築の試み / Constructing a knowledge graph for information sys...
情報システム障害解析のための知識グラフ構築の試み / Constructing a knowledge graph for information sys...情報システム障害解析のための知識グラフ構築の試み / Constructing a knowledge graph for information sys...
情報システム障害解析のための知識グラフ構築の試み / Constructing a knowledge graph for information sys...
Shinji Takao
 
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "
Deep Learning JP
 
NVIDIA Seminar ディープラーニングによる画像認識と応用事例
NVIDIA Seminar ディープラーニングによる画像認識と応用事例NVIDIA Seminar ディープラーニングによる画像認識と応用事例
NVIDIA Seminar ディープラーニングによる画像認識と応用事例
Takayoshi Yamashita
 
BERTによる文書系AIの取り組みと、Azureを用いたテーブルデータの説明性実現！
BERTによる文書系AIの取り組みと、Azureを用いたテーブルデータの説明性実現！BERTによる文書系AIの取り組みと、Azureを用いたテーブルデータの説明性実現！
BERTによる文書系AIの取り組みと、Azureを用いたテーブルデータの説明性実現！
Deep Learning Lab（ディープラーニング・ラボ）
 
翻訳精度の最大化による同時音声翻訳のための文分割法 (NLP2014)
翻訳精度の最大化による同時音声翻訳のための文分割法 (NLP2014)翻訳精度の最大化による同時音声翻訳のための文分割法 (NLP2014)
翻訳精度の最大化による同時音声翻訳のための文分割法 (NLP2014)
Yusuke Oda
 
ダウンヒルシンプレックス法について解説
ダウンヒルシンプレックス法について解説ダウンヒルシンプレックス法について解説
ダウンヒルシンプレックス法について解説
Kenyu Uehara
 
[DL輪読会]EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
[DL輪読会]EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning[DL輪読会]EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
[DL輪読会]EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
Deep Learning JP
 
AI勉強会用スライド
AI勉強会用スライドAI勉強会用スライド
AI勉強会用スライド
harmonylab
 
CycleGANで顔写真をアニメ調に変換する
CycleGANで顔写真をアニメ調に変換するCycleGANで顔写真をアニメ調に変換する
CycleGANで顔写真をアニメ調に変換する
meownoisy
 
[DL輪読会]Understanding Black-box Predictions via Influence Functions
[DL輪読会]Understanding Black-box Predictions via Influence Functions [DL輪読会]Understanding Black-box Predictions via Influence Functions
[DL輪読会]Understanding Black-box Predictions via Influence Functions
Deep Learning JP
 
【DL輪読会】StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
【DL輪読会】StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery【DL輪読会】StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
【DL輪読会】StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Deep Learning JP
 
[DL輪読会]Causality Inspired Representation Learning for Domain Generalization
[DL輪読会]Causality Inspired Representation Learning for Domain Generalization[DL輪読会]Causality Inspired Representation Learning for Domain Generalization
[DL輪読会]Causality Inspired Representation Learning for Domain Generalization
Deep Learning JP
 
Link prediction with the linkpred tool
Link prediction with the linkpred toolLink prediction with the linkpred tool
Link prediction with the linkpred tool
Raf Guns
 
[DL Hacks]CelebAをNumPyで保存してみた
[DL Hacks]CelebAをNumPyで保存してみた[DL Hacks]CelebAをNumPyで保存してみた
[DL Hacks]CelebAをNumPyで保存してみた
Deep Learning JP
 
[DLHacks]StyleGANとBigGANのStyle mixing, morphing
[DLHacks]StyleGANとBigGANのStyle mixing, morphing[DLHacks]StyleGANとBigGANのStyle mixing, morphing
[DLHacks]StyleGANとBigGANのStyle mixing, morphing
Deep Learning JP
 
[DL輪読会] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
[DL輪読会] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields [DL輪読会] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
[DL輪読会] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
Deep Learning JP
 
研究発表のやり方
研究発表のやり方研究発表のやり方
研究発表のやり方
Akinori Ito
 
【DL輪読会】Scaling Laws for Neural Language Models
【DL輪読会】Scaling Laws for Neural Language Models【DL輪読会】Scaling Laws for Neural Language Models
【DL輪読会】Scaling Laws for Neural Language Models
Deep Learning JP
 
論文紹介「A Perspective View and Survey of Meta-Learning」
論文紹介「A Perspective View and Survey of Meta-Learning」論文紹介「A Perspective View and Survey of Meta-Learning」
論文紹介「A Perspective View and Survey of Meta-Learning」
Kota Matsui
 
【DL輪読会】"A Generalist Agent"
【DL輪読会】"A Generalist Agent"【DL輪読会】"A Generalist Agent"
【DL輪読会】"A Generalist Agent"
Deep Learning JP
 

What's hot (20)

情報システム障害解析のための知識グラフ構築の試み / Constructing a knowledge graph for information sys...
情報システム障害解析のための知識グラフ構築の試み / Constructing a knowledge graph for information sys...情報システム障害解析のための知識グラフ構築の試み / Constructing a knowledge graph for information sys...
情報システム障害解析のための知識グラフ構築の試み / Constructing a knowledge graph for information sys...
 
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "
【DL輪読会】"Language Instructed Reinforcement Learning for Human-AI Coordination "
 
NVIDIA Seminar ディープラーニングによる画像認識と応用事例
NVIDIA Seminar ディープラーニングによる画像認識と応用事例NVIDIA Seminar ディープラーニングによる画像認識と応用事例
NVIDIA Seminar ディープラーニングによる画像認識と応用事例
 
BERTによる文書系AIの取り組みと、Azureを用いたテーブルデータの説明性実現！
BERTによる文書系AIの取り組みと、Azureを用いたテーブルデータの説明性実現！BERTによる文書系AIの取り組みと、Azureを用いたテーブルデータの説明性実現！
BERTによる文書系AIの取り組みと、Azureを用いたテーブルデータの説明性実現！
 
翻訳精度の最大化による同時音声翻訳のための文分割法 (NLP2014)
翻訳精度の最大化による同時音声翻訳のための文分割法 (NLP2014)翻訳精度の最大化による同時音声翻訳のための文分割法 (NLP2014)
翻訳精度の最大化による同時音声翻訳のための文分割法 (NLP2014)
 
ダウンヒルシンプレックス法について解説
ダウンヒルシンプレックス法について解説ダウンヒルシンプレックス法について解説
ダウンヒルシンプレックス法について解説
 
[DL輪読会]EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
[DL輪読会]EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning[DL輪読会]EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
[DL輪読会]EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
 
AI勉強会用スライド
AI勉強会用スライドAI勉強会用スライド
AI勉強会用スライド
 
CycleGANで顔写真をアニメ調に変換する
CycleGANで顔写真をアニメ調に変換するCycleGANで顔写真をアニメ調に変換する
CycleGANで顔写真をアニメ調に変換する
 
[DL輪読会]Understanding Black-box Predictions via Influence Functions
[DL輪読会]Understanding Black-box Predictions via Influence Functions [DL輪読会]Understanding Black-box Predictions via Influence Functions
[DL輪読会]Understanding Black-box Predictions via Influence Functions
 
【DL輪読会】StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
【DL輪読会】StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery【DL輪読会】StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
【DL輪読会】StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
 
[DL輪読会]Causality Inspired Representation Learning for Domain Generalization
[DL輪読会]Causality Inspired Representation Learning for Domain Generalization[DL輪読会]Causality Inspired Representation Learning for Domain Generalization
[DL輪読会]Causality Inspired Representation Learning for Domain Generalization
 
Link prediction with the linkpred tool
Link prediction with the linkpred toolLink prediction with the linkpred tool
Link prediction with the linkpred tool
 
[DL Hacks]CelebAをNumPyで保存してみた
[DL Hacks]CelebAをNumPyで保存してみた[DL Hacks]CelebAをNumPyで保存してみた
[DL Hacks]CelebAをNumPyで保存してみた
 
[DLHacks]StyleGANとBigGANのStyle mixing, morphing
[DLHacks]StyleGANとBigGANのStyle mixing, morphing[DLHacks]StyleGANとBigGANのStyle mixing, morphing
[DLHacks]StyleGANとBigGANのStyle mixing, morphing
 
[DL輪読会] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
[DL輪読会] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields [DL輪読会] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
[DL輪読会] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
 
研究発表のやり方
研究発表のやり方研究発表のやり方
研究発表のやり方
 
【DL輪読会】Scaling Laws for Neural Language Models
【DL輪読会】Scaling Laws for Neural Language Models【DL輪読会】Scaling Laws for Neural Language Models
【DL輪読会】Scaling Laws for Neural Language Models
 
論文紹介「A Perspective View and Survey of Meta-Learning」
論文紹介「A Perspective View and Survey of Meta-Learning」論文紹介「A Perspective View and Survey of Meta-Learning」
論文紹介「A Perspective View and Survey of Meta-Learning」
 
【DL輪読会】"A Generalist Agent"
【DL輪読会】"A Generalist Agent"【DL輪読会】"A Generalist Agent"
【DL輪読会】"A Generalist Agent"
 

Similar to Building, Growing and Serving Large Knowledge Graphs with Human-in-the-Loop

Information Technology in Industry(ITII) - November Issue 2018
Information Technology in Industry(ITII) - November Issue 2018Information Technology in Industry(ITII) - November Issue 2018
Information Technology in Industry(ITII) - November Issue 2018
ITIIIndustries
 
Digital Twin and Smart Spaces
Digital Twin and Smart Spaces Digital Twin and Smart Spaces
Digital Twin and Smart Spaces
SANGHEE SHIN
 
Cloud Services for Repositories
Cloud Services for RepositoriesCloud Services for Repositories
Cloud Services for Repositories
Eduserv
 
Compositional AI: Fusion of AI/ML Services
Compositional AI: Fusion of AI/ML ServicesCompositional AI: Fusion of AI/ML Services
Compositional AI: Fusion of AI/ML Services
Debmalya Biswas
 
Bridging the Gap
Bridging the GapBridging the Gap
Bridging the Gap
Dickson Lukose
 
A New Paradigm on Analytic-Driven Information and Automation V2.pdf
A New Paradigm on Analytic-Driven Information and Automation V2.pdfA New Paradigm on Analytic-Driven Information and Automation V2.pdf
A New Paradigm on Analytic-Driven Information and Automation V2.pdf
ArmyTrilidiaDevegaSK
 
Rethinking enterprise architecture for DevOps, Agile, and cloud native organi...
Rethinking enterprise architecture for DevOps, Agile, and cloud native organi...Rethinking enterprise architecture for DevOps, Agile, and cloud native organi...
Rethinking enterprise architecture for DevOps, Agile, and cloud native organi...
Michael Coté
 
Stary2020_Chapter_TheInternet-of-BehaviorAsOrganRG.pdf
Stary2020_Chapter_TheInternet-of-BehaviorAsOrganRG.pdfStary2020_Chapter_TheInternet-of-BehaviorAsOrganRG.pdf
Stary2020_Chapter_TheInternet-of-BehaviorAsOrganRG.pdf
Hải Quân
 
Summit Australia 2019 - Supercharge PowerPlatform with AI - Dipankar Bhattach...
Summit Australia 2019 - Supercharge PowerPlatform with AI - Dipankar Bhattach...Summit Australia 2019 - Supercharge PowerPlatform with AI - Dipankar Bhattach...
Summit Australia 2019 - Supercharge PowerPlatform with AI - Dipankar Bhattach...
Andrew Ly
 
Koneksys - Offering Services to Connect Data using the Data Web
Koneksys - Offering Services to Connect Data using the Data WebKoneksys - Offering Services to Connect Data using the Data Web
Koneksys - Offering Services to Connect Data using the Data Web
Koneksys
 
Ai open powermeetupmarch25th
Ai open powermeetupmarch25thAi open powermeetupmarch25th
Ai open powermeetupmarch25th
IBM
 
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystemStrata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Shirshanka Das
 
Architecting for change: LinkedIn's new data ecosystem
Architecting for change: LinkedIn's new data ecosystemArchitecting for change: LinkedIn's new data ecosystem
Architecting for change: LinkedIn's new data ecosystem
Yael Garten
 
Kazakhstan digital media_at_svl 20191018 v5
Kazakhstan digital media_at_svl 20191018 v5Kazakhstan digital media_at_svl 20191018 v5
Kazakhstan digital media_at_svl 20191018 v5
ISSIP
 
Intel 20180608 v2
Intel 20180608 v2Intel 20180608 v2
Intel 20180608 v2
ISSIP
 
Choosing the Right Technologies A Guide to Frameworks and Tools for Web App D...
Choosing the Right Technologies A Guide to Frameworks and Tools for Web App D...Choosing the Right Technologies A Guide to Frameworks and Tools for Web App D...
Choosing the Right Technologies A Guide to Frameworks and Tools for Web App D...
BitCot
 
Ai open powermeetupmarch25th
Ai open powermeetupmarch25thAi open powermeetupmarch25th
Ai open powermeetupmarch25th
IBM
 
Ai open powermeetupmarch25th
Ai open powermeetupmarch25thAi open powermeetupmarch25th
Ai open powermeetupmarch25th
IBM
 
Ajay ohri Resume
Ajay ohri ResumeAjay ohri Resume
Ajay ohri Resume
Ajay Ohri
 
20240425 PSU_Spring_2024 AI_Digital_Twins AI_Collab White Paper.pdf
20240425 PSU_Spring_2024 AI_Digital_Twins AI_Collab White Paper.pdf20240425 PSU_Spring_2024 AI_Digital_Twins AI_Collab White Paper.pdf
20240425 PSU_Spring_2024 AI_Digital_Twins AI_Collab White Paper.pdf
International Society of Service Innovation Professionals
 

Similar to Building, Growing and Serving Large Knowledge Graphs with Human-in-the-Loop (20)

Information Technology in Industry(ITII) - November Issue 2018
Information Technology in Industry(ITII) - November Issue 2018Information Technology in Industry(ITII) - November Issue 2018
Information Technology in Industry(ITII) - November Issue 2018
 
Digital Twin and Smart Spaces
Digital Twin and Smart Spaces Digital Twin and Smart Spaces
Digital Twin and Smart Spaces
 
Cloud Services for Repositories
Cloud Services for RepositoriesCloud Services for Repositories
Cloud Services for Repositories
 
Compositional AI: Fusion of AI/ML Services
Compositional AI: Fusion of AI/ML ServicesCompositional AI: Fusion of AI/ML Services
Compositional AI: Fusion of AI/ML Services
 
Bridging the Gap
Bridging the GapBridging the Gap
Bridging the Gap
 
A New Paradigm on Analytic-Driven Information and Automation V2.pdf
A New Paradigm on Analytic-Driven Information and Automation V2.pdfA New Paradigm on Analytic-Driven Information and Automation V2.pdf
A New Paradigm on Analytic-Driven Information and Automation V2.pdf
 
Rethinking enterprise architecture for DevOps, Agile, and cloud native organi...
Rethinking enterprise architecture for DevOps, Agile, and cloud native organi...Rethinking enterprise architecture for DevOps, Agile, and cloud native organi...
Rethinking enterprise architecture for DevOps, Agile, and cloud native organi...
 
Stary2020_Chapter_TheInternet-of-BehaviorAsOrganRG.pdf
Stary2020_Chapter_TheInternet-of-BehaviorAsOrganRG.pdfStary2020_Chapter_TheInternet-of-BehaviorAsOrganRG.pdf
Stary2020_Chapter_TheInternet-of-BehaviorAsOrganRG.pdf
 
Summit Australia 2019 - Supercharge PowerPlatform with AI - Dipankar Bhattach...
Summit Australia 2019 - Supercharge PowerPlatform with AI - Dipankar Bhattach...Summit Australia 2019 - Supercharge PowerPlatform with AI - Dipankar Bhattach...
Summit Australia 2019 - Supercharge PowerPlatform with AI - Dipankar Bhattach...
 
Koneksys - Offering Services to Connect Data using the Data Web
Koneksys - Offering Services to Connect Data using the Data WebKoneksys - Offering Services to Connect Data using the Data Web
Koneksys - Offering Services to Connect Data using the Data Web
 
Ai open powermeetupmarch25th
Ai open powermeetupmarch25thAi open powermeetupmarch25th
Ai open powermeetupmarch25th
 
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystemStrata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
 
Architecting for change: LinkedIn's new data ecosystem
Architecting for change: LinkedIn's new data ecosystemArchitecting for change: LinkedIn's new data ecosystem
Architecting for change: LinkedIn's new data ecosystem
 
Kazakhstan digital media_at_svl 20191018 v5
Kazakhstan digital media_at_svl 20191018 v5Kazakhstan digital media_at_svl 20191018 v5
Kazakhstan digital media_at_svl 20191018 v5
 
Intel 20180608 v2
Intel 20180608 v2Intel 20180608 v2
Intel 20180608 v2
 
Choosing the Right Technologies A Guide to Frameworks and Tools for Web App D...
Choosing the Right Technologies A Guide to Frameworks and Tools for Web App D...Choosing the Right Technologies A Guide to Frameworks and Tools for Web App D...
Choosing the Right Technologies A Guide to Frameworks and Tools for Web App D...
 
Ai open powermeetupmarch25th
Ai open powermeetupmarch25thAi open powermeetupmarch25th
Ai open powermeetupmarch25th
 
Ai open powermeetupmarch25th
Ai open powermeetupmarch25thAi open powermeetupmarch25th
Ai open powermeetupmarch25th
 
Ajay ohri Resume
Ajay ohri ResumeAjay ohri Resume
Ajay ohri Resume
 
20240425 PSU_Spring_2024 AI_Digital_Twins AI_Collab White Paper.pdf
20240425 PSU_Spring_2024 AI_Digital_Twins AI_Collab White Paper.pdf20240425 PSU_Spring_2024 AI_Digital_Twins AI_Collab White Paper.pdf
20240425 PSU_Spring_2024 AI_Digital_Twins AI_Collab White Paper.pdf
 

More from Yunyao Li

The Role of Patterns in the Era of Large Language Models
The Role of Patterns in the Era of Large Language ModelsThe Role of Patterns in the Era of Large Language Models
The Role of Patterns in the Era of Large Language Models
Yunyao Li
 
Meaning Representations for Natural Languages: Design, Models and Applications
Meaning Representations for Natural Languages: Design, Models and ApplicationsMeaning Representations for Natural Languages: Design, Models and Applications
Meaning Representations for Natural Languages: Design, Models and Applications
Yunyao Li
 
Taming the Wild West of NLP
Taming the Wild West of NLPTaming the Wild West of NLP
Taming the Wild West of NLP
Yunyao Li
 
Towards Deep Table Understanding
Towards Deep Table UnderstandingTowards Deep Table Understanding
Towards Deep Table Understanding
Yunyao Li
 
Explainability for Natural Language Processing
Explainability for Natural Language ProcessingExplainability for Natural Language Processing
Explainability for Natural Language Processing
Yunyao Li
 
Explainability for Natural Language Processing
Explainability for Natural Language ProcessingExplainability for Natural Language Processing
Explainability for Natural Language Processing
Yunyao Li
 
Human in the Loop AI for Building Knowledge Bases
Human in the Loop AI for Building Knowledge Bases Human in the Loop AI for Building Knowledge Bases
Human in the Loop AI for Building Knowledge Bases
Yunyao Li
 
Towards Universal Language Understanding
Towards Universal Language UnderstandingTowards Universal Language Understanding
Towards Universal Language Understanding
Yunyao Li
 
Explainability for Natural Language Processing
Explainability for Natural Language ProcessingExplainability for Natural Language Processing
Explainability for Natural Language Processing
Yunyao Li
 
Towards Universal Language Understanding (2020 version)
Towards Universal Language Understanding (2020 version)Towards Universal Language Understanding (2020 version)
Towards Universal Language Understanding (2020 version)
Yunyao Li
 
Towards Universal Semantic Understanding of Natural Languages
Towards Universal Semantic Understanding of Natural LanguagesTowards Universal Semantic Understanding of Natural Languages
Towards Universal Semantic Understanding of Natural Languages
Yunyao Li
 
An In-depth Analysis of the Effect of Text Normalization in Social Media
An In-depth Analysis of the Effect of Text Normalization in Social MediaAn In-depth Analysis of the Effect of Text Normalization in Social Media
An In-depth Analysis of the Effect of Text Normalization in Social Media
Yunyao Li
 
Exploiting Structure in Representation of Named Entities using Active Learning
Exploiting Structure in Representation of Named Entities using Active LearningExploiting Structure in Representation of Named Entities using Active Learning
Exploiting Structure in Representation of Named Entities using Active Learning
Yunyao Li
 
K-SRL: Instance-based Learning for Semantic Role Labeling
K-SRL: Instance-based Learning for Semantic Role LabelingK-SRL: Instance-based Learning for Semantic Role Labeling
K-SRL: Instance-based Learning for Semantic Role Labeling
Yunyao Li
 
Coling poster
Coling posterColing poster
Coling poster
Yunyao Li
 
Coling demo
Coling demoColing demo
Coling demo
Yunyao Li
 
Natural Language Data Management and Interfaces: Recent Development and Open ...
Natural Language Data Management and Interfaces: Recent Development and Open ...Natural Language Data Management and Interfaces: Recent Development and Open ...
Natural Language Data Management and Interfaces: Recent Development and Open ...
Yunyao Li
 
Polyglot: Multilingual Semantic Role Labeling with Unified Labels
Polyglot: Multilingual Semantic Role Labeling with Unified LabelsPolyglot: Multilingual Semantic Role Labeling with Unified Labels
Polyglot: Multilingual Semantic Role Labeling with Unified Labels
Yunyao Li
 
Transparent Machine Learning for Information Extraction: State-of-the-Art and...
Transparent Machine Learning for Information Extraction: State-of-the-Art and...Transparent Machine Learning for Information Extraction: State-of-the-Art and...
Transparent Machine Learning for Information Extraction: State-of-the-Art and...
Yunyao Li
 
The Power of Declarative Analytics
The Power of Declarative AnalyticsThe Power of Declarative Analytics
The Power of Declarative Analytics
Yunyao Li
 

More from Yunyao Li (20)

The Role of Patterns in the Era of Large Language Models
The Role of Patterns in the Era of Large Language ModelsThe Role of Patterns in the Era of Large Language Models
The Role of Patterns in the Era of Large Language Models
 
Meaning Representations for Natural Languages: Design, Models and Applications
Meaning Representations for Natural Languages: Design, Models and ApplicationsMeaning Representations for Natural Languages: Design, Models and Applications
Meaning Representations for Natural Languages: Design, Models and Applications
 
Taming the Wild West of NLP
Taming the Wild West of NLPTaming the Wild West of NLP
Taming the Wild West of NLP
 
Towards Deep Table Understanding
Towards Deep Table UnderstandingTowards Deep Table Understanding
Towards Deep Table Understanding
 
Explainability for Natural Language Processing
Explainability for Natural Language ProcessingExplainability for Natural Language Processing
Explainability for Natural Language Processing
 
Explainability for Natural Language Processing
Explainability for Natural Language ProcessingExplainability for Natural Language Processing
Explainability for Natural Language Processing
 
Human in the Loop AI for Building Knowledge Bases
Human in the Loop AI for Building Knowledge Bases Human in the Loop AI for Building Knowledge Bases
Human in the Loop AI for Building Knowledge Bases
 
Towards Universal Language Understanding
Towards Universal Language UnderstandingTowards Universal Language Understanding
Towards Universal Language Understanding
 
Explainability for Natural Language Processing
Explainability for Natural Language ProcessingExplainability for Natural Language Processing
Explainability for Natural Language Processing
 
Towards Universal Language Understanding (2020 version)
Towards Universal Language Understanding (2020 version)Towards Universal Language Understanding (2020 version)
Towards Universal Language Understanding (2020 version)
 
Towards Universal Semantic Understanding of Natural Languages
Towards Universal Semantic Understanding of Natural LanguagesTowards Universal Semantic Understanding of Natural Languages
Towards Universal Semantic Understanding of Natural Languages
 
An In-depth Analysis of the Effect of Text Normalization in Social Media
An In-depth Analysis of the Effect of Text Normalization in Social MediaAn In-depth Analysis of the Effect of Text Normalization in Social Media
An In-depth Analysis of the Effect of Text Normalization in Social Media
 
Exploiting Structure in Representation of Named Entities using Active Learning
Exploiting Structure in Representation of Named Entities using Active LearningExploiting Structure in Representation of Named Entities using Active Learning
Exploiting Structure in Representation of Named Entities using Active Learning
 
K-SRL: Instance-based Learning for Semantic Role Labeling
K-SRL: Instance-based Learning for Semantic Role LabelingK-SRL: Instance-based Learning for Semantic Role Labeling
K-SRL: Instance-based Learning for Semantic Role Labeling
 
Coling poster
Coling posterColing poster
Coling poster
 
Coling demo
Coling demoColing demo
Coling demo
 
Natural Language Data Management and Interfaces: Recent Development and Open ...
Natural Language Data Management and Interfaces: Recent Development and Open ...Natural Language Data Management and Interfaces: Recent Development and Open ...
Natural Language Data Management and Interfaces: Recent Development and Open ...
 
Polyglot: Multilingual Semantic Role Labeling with Unified Labels
Polyglot: Multilingual Semantic Role Labeling with Unified LabelsPolyglot: Multilingual Semantic Role Labeling with Unified Labels
Polyglot: Multilingual Semantic Role Labeling with Unified Labels
 
Transparent Machine Learning for Information Extraction: State-of-the-Art and...
Transparent Machine Learning for Information Extraction: State-of-the-Art and...Transparent Machine Learning for Information Extraction: State-of-the-Art and...
Transparent Machine Learning for Information Extraction: State-of-the-Art and...
 
The Power of Declarative Analytics
The Power of Declarative AnalyticsThe Power of Declarative Analytics
The Power of Declarative Analytics
 

Recently uploaded

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Product School
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
OnBoard
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
Welocme to ViralQR, your best QR code generator.
Welocme to ViralQR, your best QR code generator.Welocme to ViralQR, your best QR code generator.
Welocme to ViralQR, your best QR code generator.
ViralQR
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
sonjaschweigert1
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
Free Complete Python - A step towards Data Science
Free Complete Python - A step towards Data ScienceFree Complete Python - A step towards Data Science
Free Complete Python - A step towards Data Science
RinaMondal9
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
DianaGray10
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
Dorra BARTAGUIZ
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
Prayukth K V
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
Ralf Eggert
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
Pierluigi Pugliese
 

Recently uploaded (20)

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
Welocme to ViralQR, your best QR code generator.
Welocme to ViralQR, your best QR code generator.Welocme to ViralQR, your best QR code generator.
Welocme to ViralQR, your best QR code generator.
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
Free Complete Python - A step towards Data Science
Free Complete Python - A step towards Data ScienceFree Complete Python - A step towards Data Science
Free Complete Python - A step towards Data Science
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
 

Building, Growing and Serving Large Knowledge Graphs with Human-in-the-Loop

  • 1. Yunyao Li HILDA @ SIGMOD’23 | Apple Inc. | June 18, 2023 Building, Growing and Serving Large Knowledge Graphs with Human-in-the-Loop
  • 2. Knowledge Bases Image Source: https://www.csee.umbc.edu/courses/graduate/691/fall22/kg/
  • 3. Example: Financial Content Knowledge Base Financial Reports Ontology [VLDB’2017] Creation and Interaction with Large-scale Domain-Specific Knowledge Bases. XML Knowledge Extraction Overall Architecture: A Simplified View Linking Fusion KG Construction Transforming >31,000 companies 439 industries ~170,000 insiders ~100 millions financial metrics ~22,000 industry KPIs Financial Content KB KG Services QA APIs
  • 4. Example: Saga Structured Knowledge Sources Real-time Sources Ontology Unstructured Knowledge Sources Linking Fusion KG Construction KG Knowledge Extraction KG Services QA Semantic Annotation … … Embedding Services [SIGMOD’23] Growing and Serving Large Open-domain Knowledge Graphs. [SIGMOD’22] Saga: A Platform for Continuous Construction and Serving of Knowledge at Scale Transforming Overall Architecture: A Simplified View
  • 5. Human-in-the-Loop Throughout the Entire Life Cycle of KG Construction, Growth, and Services Data Labeling Development Deployment
  • 6. Human-in-the-Loop Throughout the Entire Life Cycle of KG Construction, Growth, and Services Data Labeling Development Deployment Learner raw data labeled data 1. Improve quality 2. Increase efficiency 3. Decrease skill requirements
  • 7. Example 1: Scale Fact Collection Missing / stale facts Missing Facts Query Synthesizer QA System candidate facts Baseline New Facts
  • 8. Example 1: Scale Fact Collection Missing / stale facts Missing Facts Query Synthesizer QA System candidate facts Baseline New Facts Query-by-Committee Missing Facts Query Synthesizer QA System candidate facts New Facts QA System Q1 QA System … … … … … Qn QbC Selector AnswerSet1 AnswerSetn [EMNLP-DaSH’2022] Improving Human Annotation Effectiveness for Fact Collection by Identifying the Most Relevant Answers Success Rate fact collection 25%
  • 9. Example 1: Scale Fact Collection Missing / stale facts Missing Facts Query Synthesizer QA System candidate facts Baseline New Facts Open Domain Knowledge Extraction [SIGMOD’23] Growing and Serving Large Open-domain Knowledge Graphs. Throughput vs. manual fact collection >100x Missing Facts Query Synthesizer Web Search candidate facts w/ lower-confidence New Facts Knowledge Extractor Fact Corroboration
  • 10. Example 2: Semantic Role Labeling SRL
  • 11. Example 2a: Crowd-in-the-Loop Curation An hybrid approach Corpus raw data Corpus predicated annotations Annotation Task Corpus curated annotations Task Router Difficult tasks are curated by experts Easier tasks are curated by crowd [EMNLP’17] CROWD-IN-THE-LOOP: A Hybrid Approach for Annotating Semantic Roles
  • 12. Example 2a: Effectiveness of Crowd-in-the-Loop 9% F1 vs. SRL model ↑ 66.4% Expert efforts ↓
  • 13. Example 2b: Better Workflow Performs Ever Better vs. SRL model ↑ Expert efforts ↓ 10% F1 vs. SRL model ↑ 87.3% Expert efforts ↓ Filter unlikely options Select from likely options Expert resolve hard cases [EMNLP’20 (Finding)] A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels
  • 14. Human-in-the-Loop Throughout the Entire Life Cycle of KG construction, growth, and services Data Labeling Development Deployment Learner Scale data labeling raw data labeled data IDE Better IDE for model building
  • 15. Different Tooling for Different Users Full-fledged IDE AI Engineers AI Engineers/Data Scientists Visual IDE [ACL’12] WizIE: A Best Practices Guided Development Environment for Information Extraction [CHI’13] I can do text analytics!: designing development tools for novice developers [VLDB’15] VINERy: A Visual IDE for Information Extraction [KDD’19] Declarative Text Understanding with SystemT. (hands-on tutorial) Entity Extraction in AIOps https://www.ibm.com/cloud/blog/entity-extraction-in-aiops IBM InfoSphere BigInsights Text Analytics Eclipse Tooling IBM Watson Knowledge Studio. Advanced Rule Editor http://ibm.biz/VineryIE
  • 16. Human-in-the-Loop Throughout the Entire Life Cycle of KG construction, growth, and services Data Labeling Development Deployment Learner Scale data labeling raw data labeled data IDE Better IDE for model building Learner Human-machine co-creation
  • 17. Transparent Linguistic Models for Contract Understanding Watson Discovery Content Intelligence [NAACL’21] Development of an Enterprise-Grade Contract Understanding System, (Industry Track)
  • 18. HEIDL: Human & Machine Co-Creation via Neural-Symbolic AI [ACL’19] HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop. [EMNLP’20] Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification In use for major IBM customer engagements Raises the abstraction level for domain experts to interact with
  • 19. HEIDL: Human & Machine Co-Creation via Neural-Symbolic AI [ACL’19] HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop. [EMNLP’20] Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification In use for major IBM customer engagements Raises the abstraction level for domain experts to interact with
  • 20. Human-in-the-Loop Throughout the Entire Life Cycle of KG construction, growth, and services Data Labeling Development Deployment Learner Scale data labeling raw data labeled data IDE Better IDE for model building Learner Human-machine co-creation Learner Curb data hunger with interactive learning
  • 21. Case 1: Example-Driven Extraction Via pattern induction [CHI’17] SEER: Auto-Generating Information Extraction Rules from User-Specified Examples [SIGMOD’17] Synthesizing Extraction Rules from User Examples with SEER. SIGMOD’2017 [AAAI’22 (demo)] InteractEva: A Simulation-based Evaluation Framework for Interactive AI Systems [AAAI’22] A Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application. IBM Watson Discovery (Beta in Plus since Oct. 2021) http://ibm.biz/SEER_IE, https://ibm.biz/WDSPressReleaseNov
  • 22. Case 1: Example-Driven Extraction Via pattern induction [CHI’17] SEER: Auto-Generating Information Extraction Rules from User-Specified Examples [SIGMOD’17] Synthesizing Extraction Rules from User Examples with SEER. SIGMOD’2017 [AAAI’22 (demo)] InteractEva: A Simulation-based Evaluation Framework for Interactive AI Systems [AAAI’22] A Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application. IBM Watson Discovery (Beta in Plus since Oct. 2021) http://ibm.biz/SEER_IE, https://ibm.biz/WDSPressReleaseNov
  • 23. Case 1: Example-Driven Extraction Via pattern induction [CHI’17] SEER: Auto-Generating Information Extraction Rules from User-Specified Examples [SIGMOD’17] Synthesizing Extraction Rules from User Examples with SEER. SIGMOD’2017 [AAAI’22 (demo)] InteractEva: A Simulation-based Evaluation Framework for Interactive AI Systems [AAAI’22] A Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application. IBM Watson Discovery (Beta in Plus since Oct. 2021) http://ibm.biz/SEER_IE, https://ibm.biz/WDSPressReleaseNov
  • 24. Case 1: Example-Driven Extraction Via pattern induction [CHI’17] SEER: Auto-Generating Information Extraction Rules from User-Specified Examples [SIGMOD’17] Synthesizing Extraction Rules from User Examples with SEER. SIGMOD’2017 [AAAI’22 (demo)] InteractEva: A Simulation-based Evaluation Framework for Interactive AI Systems [AAAI’22] A Simulation-Based Evaluation Framework for Interactive AI Systems and Its Application. IBM Watson Discovery (Beta in Plus since Oct. 2021) http://ibm.biz/SEER_IE, https://ibm.biz/WDSPressReleaseNov
  • 25. Case 2: Entity Normalization & Variant Generation Learning Structured Representations Capture Entity Semantic Structure [COLING’2018] Exploiting Structure in Representation of Named Entities using Active Learning. [ICDE’2018] LUSTRE: An Interactive System for Entity Structured Representation and Variant Generation. Generated normalizers for Watson Discovery [AAAI’2020] PARTNER: Human-in-the-Loop Entity Name Understanding with Deep Learning. [EMNLP’2020] Learning Structured Representations of Entity Names using Active Learning and Weak Supervision. “Bank of America N.A.” “Bank of America National Association” Synthesizing Normalization and Variant Generation Functions
  • 26. Case 3: Deep Document Understanding Document Ingestion [WACV 2021] Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context. [AAAI’21] KAAPA: Knowledge Aware Answers from PDF Analysis. [ACL-CORD-19’21] CORD-19: The COVID-19 Open Research Dataset Bringing IBM NLP capabilities to the CORD-19 Dataset. http://ibm.biz/CORD19-IBM IBM Watson Discovery JSON/HTML Wide Variety in PDF Tables Table with graphic lines Table with visual clues only Complex table with multi-row/column headers Table interleaved with text and charts
  • 27. Case 3: TableLab TableLab: Easy Customization via Adaptive Deep Learning [IUI’2021] TableLab: An Interactive Table Extraction System with Adaptive Deep Learning.
  • 28. Case 3: Deep Document Understanding TableLab: Easy Customization via Adaptive Deep Learning [IUI’2021] TableLab: An Interactive Table Extraction System with Adaptive Deep Learning.
  • 29. Case 3: Customization vis TableLab Table Boundary Detection Preliminary Results Method CEDAR EDGAR Invoices Appraisals Health Docs GTE 0.94 0.84 0.47 0.85 0.93 GTE with Retraining 0.96 0.91 0.92 0.96 0.98 Method CEDAR EDGAR Invoices Appraisals Health Docs GTE 0.88 0.62 0.42 0.71 0.55 GTE with Retraining 0.90 0.82 0.68 0.90 0.77 Cell Adjacency Detection Dataset 20 pages with tables per category: 10 for retraining, 10 for testing Evaluation Metric F1 metric for Table Boundary and Cell Adjacency as de fi ned in [1] [1] Göbel et al. “A Methodology for Evaluating Algorithms for Table Understanding in PDF Documents”. DocEng '12
  • 30. Case 4: Label Sleuth An open-source no-code system for text annotation and building text classifiers [EMNLP’2022] Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours https://www.label-sleuth.org 1. From task definition to working model in hours! 2. Extensible backend to integrate new model architectures or active learning techniques
  • 31. Human-in-the-Loop Throughout the Entire Life Cycle of KG construction, growth, and services Data Labeling Development Deployment Learner Scale data labeling raw data labeled data IDE Better IDE for model building Learner Human-machine co-creation Learner Curb data hunger with interactive learning AutoML Scale model building via AutoML
  • 32. AutoAI for Text AutoText [AAAI’21] AutoText: An End-to-End AutoAI Framework for Text. [NeurIPS 2022] AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning. IBM Developer API https://developer.ibm.com/learningpaths/get-started-autoai-for-text-api Example Use Case: Scale ML Product Model for Text Classification >30% Reduction in combined training and prediction time Auto weight tuning & HPO >10x Speed-up in training at comparable quality Auto classifier selection
  • 33. Human-in-the-Loop Throughout the Entire Life Cycle of KG construction, growth, and services Data Labeling Development Deployment Learner Scale data labeling raw data labeled data IDE Better IDE for model building Learner Human-machine co-creation Learner Curb data hunger with interactive learning AutoML Scale model building via AutoML
  • 34. Human-in-the-Loop Throughout the Entire Life Cycle of KG construction, growth, and services Data Labeling Development Deployment Learner Scale data labeling raw data labeled data IDE Better IDE for model building Learner Human-machine co-creation Learner Curb data hunger with interactive learning AutoML Scale model building via AutoML Query Log Tickets User feedback influence the entire life cycle
  • 35. Quality Evaluation 1. Measure what matters for end users 2. Identify the root cause of failures 3. Track improvements in individual components as they evolve The key requirements - Who won the Paris Paris, France Paris Masters
  • 36. Overall Evaluation Framework A Human-in-the-Loop Process Annotation Quality Metrics Dataset Collection Tooling and annotation guidelines for graders Evaluation Human in the loop to annotate/ grade queries Logs Synthetic Queries Knowledge Graph Metrics End to End Metrics Query Understanding Metrics
  • 37. Visual Tooling of Metrics and Loss Buckets - Example Errors: - Entity Prediction Error: “Who won Paris” (Paris Masters/Paris–Roubaix) - Missing Fact: “When is the oscars in 2026” - Fact is not present because date/location is not published yet) - Unrecognized Entity in KG: ”Who is princess noor horse” Facilitate Opportunity Analysis
  • 38. Visual Diagnosis [DaSH@KDD. 2020] WhyFlow: Explaining Errors in Data Flows Interactively.
  • 39. ModelLens Visual interactive tool for model improvement [CSCW’19] ModelLens: An Interactive System to Support the Model Improvement Practices of Data Science Teams.
  • 40. So how will EVERYTHING change with LLMs? Many exciting challenges and opportunities
  • 41. Thanks! IBM (including interns): • Shivakumar Vaithyanathan • Lucian Popa • Ron Fagin • Sriram Raghavan • Rajasekar Krishnamurthy • Fred Reiss • Laura Chiticariu • Benny Kimelfeld • Mauricio Hernadez • Eser Kandogan • Huaiyu Zhu • Kun Qian • Dakuo Wang • Maeda Hanafi Many amazing collaborators and interns … Apple (including interns): • Ihab Ilyas • Theodoros Rekatsinas • Umar Farooq Minhas • Ali Mousavi • Jefferey Pound • Anil Pacaci • Shihabur R. Chowdhury • Hongyu Ren • Jason Mohoney • Kun Qian • Yiwen Sun • Yisi Sang • Saloni Potdar • … … Universities: • Azza Abouzeid (NYU-Abu Dhabi) • H. V. Jagadish (U. Of Michigan) • Fei Xia (U. Of Washington) • Kevin Chen-Chuan Chang (UIUC) • ChengXiang Zhai (UIUC) • Domenico Lembo(Sapienza University of Rome) • Dragomir R. Radev (Yale) • Jonathan K. Kummerfeld (U. Of Michigan) • Walter S. Lasecki (U. Of Michigan) • Toby Li (U. of Notre Dame) • Rishabh Iyer (UT Dallas) • Eduard C. Dragut (Temple Univ.) • … …. • Douglas Burdick’ • Alan Akbik • Nancy Wang • Prithiviraj Sen • Marina Danilevsky • Poornima Chozhiyath Raman • Sudarshan Rangarajan • Ramiya Venkatachalam • Kiran Kate • Eyal Shnarch • Ishan Jindal • Yiwei Yang • Nikita Bhutani • … ….