SlideShare a Scribd company logo

2021-11, EMNLP, Distilling Relation Embeddings from Pre-trained Language Models

Pre-trained language models have been found to capture a surprisingly rich amount of lexical knowledge, ranging from commonsense properties of everyday concepts to detailed factual knowledge about named entities. Among others, this makes it possible to distill high-quality word vectors from pre-trained language models. However, it is currently unclear to what extent it is possible to distill relation embeddings, i.e. vectors that characterize the relationship between two words. Such relation embeddings are appealing because they can, in principle, encode relational knowledge in a more fine-grained way than is possible with knowledge graphs. To obtain relation embeddings from a pre-trained language model, we encode word pairs using a (manually or automatically generated) prompt, and we fine-tune the language model such that relationally similar word pairs yield similar output vectors. We find that the resulting relation embeddings are highly competitive on analogy (unsupervised) and relation classification (supervised) benchmarks, even without any task-specific fine-tuning. Source code to reproduce our experimental results and the model checkpoints are available in the following repository: https://github.com/asahi417/relbert

1 of 24
Download to read offline
Distilling Relation Embeddings from Pre-trained
Language Models
Asahi Ushio
Jose Camacho-Collados
Steven Schockaert
https://github.com/asahi417/relbert
Distilling Relation Embeddings from Pre-trained Language Models
Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados
Language Model Understanding
2
Syntactic Knowledge
- Probing embedding: Hewitt 2019, Tenney 2019
- Probing attention weight: Clark 2020
Factual Knowledge a.k.a Language Model as a Commonsense KB
- Petroni 2019, Kassner 2020, Jiang 2020, etc
Relational Knowledge a.k.a Language Model as a Lexical Relation Reasoner
- LM fine-tuning on relation classification: Bouraoui 2019
- Vanilla LM evaluation: Ushio 2021
Distilling Relation Embeddings from Pre-trained Language Models
Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados
Language Model Understanding
3
Syntactic Knowledge
- Probing embedding: Hewitt 2019, Tenney 2019
- Probing attention weight: Clark 2020
Factual Knowledge a.k.a Language Model as a Commonsense KB
- Petroni 2019, Kassner 2020, Jiang 2020, etc
Relational Knowledge a.k.a Language Model as a Lexical Relation Reasoner
- LM fine-tuning on relation classification: Bouraoui 2019
- Vanilla LM evaluation: Ushio 2021
Distilling Relation Embeddings from Pre-trained Language Models
Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados
Language Model Understanding
4
Syntactic Knowledge
- Probing embedding: Hewitt 2019, Tenney 2019
- Probing attention weight: Clark 2020
Factual Knowledge a.k.a Language Model as a Commonsense KB
- Petroni 2019, Kassner 2020, Jiang 2020, etc
Relational Knowledge a.k.a Language Model as a Lexical Relation Reasoner
- LM fine-tuning on relation classification: Bouraoui 2019
- Vanilla LM evaluation: Ushio 2021
Can we distil relational knowledge as relation embedding?
Distilling Relation Embeddings from Pre-trained Language Models
Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados
Relation Embedding
Word Embedding Mikolov (2013)
Pair2Vec Joshi (2019)
Relative Camacho-Collados (2019)
5
King
Queen
Woman
Man
Word Embedding
RelBERT
Ad

Recommended

2022-12, EMNLP, Generative Language Models for Paragraph-Level Question Gener...
2022-12, EMNLP, Generative Language Models for Paragraph-Level Question Gener...2022-12, EMNLP, Generative Language Models for Paragraph-Level Question Gener...
2022-12, EMNLP, Generative Language Models for Paragraph-Level Question Gener...asahiushio1
 
2022-01, NLP colloquium (Japanese), Toward a Better Understanding of Relation...
2022-01, NLP colloquium (Japanese), Toward a Better Understanding of Relation...2022-01, NLP colloquium (Japanese), Toward a Better Understanding of Relation...
2022-01, NLP colloquium (Japanese), Toward a Better Understanding of Relation...asahiushio1
 
2022-10, UCL NLP meetup, Toward a Better Understanding of Relational Knowledg...
2022-10, UCL NLP meetup, Toward a Better Understanding of Relational Knowledg...2022-10, UCL NLP meetup, Toward a Better Understanding of Relational Knowledg...
2022-10, UCL NLP meetup, Toward a Better Understanding of Relational Knowledg...asahiushio1
 
2017-12, Keio University, Projection-based Regularized Dual Averaging for Sto...
2017-12, Keio University, Projection-based Regularized Dual Averaging for Sto...2017-12, Keio University, Projection-based Regularized Dual Averaging for Sto...
2017-12, Keio University, Projection-based Regularized Dual Averaging for Sto...asahiushio1
 
2017-07, Research Seminar at Keio University, Metric Perspective of Stochasti...
2017-07, Research Seminar at Keio University, Metric Perspective of Stochasti...2017-07, Research Seminar at Keio University, Metric Perspective of Stochasti...
2017-07, Research Seminar at Keio University, Metric Perspective of Stochasti...asahiushio1
 
2017-03, ICASSP, Projection-based Dual Averaging for Stochastic Sparse Optimi...
2017-03, ICASSP, Projection-based Dual Averaging for Stochastic Sparse Optimi...2017-03, ICASSP, Projection-based Dual Averaging for Stochastic Sparse Optimi...
2017-03, ICASSP, Projection-based Dual Averaging for Stochastic Sparse Optimi...asahiushio1
 
2021-04, EACL, T-NER: An All-Round Python Library for Transformer-based Named...
2021-04, EACL, T-NER: An All-Round Python Library for Transformer-based Named...2021-04, EACL, T-NER: An All-Round Python Library for Transformer-based Named...
2021-04, EACL, T-NER: An All-Round Python Library for Transformer-based Named...asahiushio1
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfmarketingartwork
 

More Related Content

Recently uploaded

commercial production of cellulase enzyme and its uses
commercial production of cellulase enzyme and its usescommercial production of cellulase enzyme and its uses
commercial production of cellulase enzyme and its usesSilpa Selvaraj
 
A galactic microquasar mimicking winged radio galaxies
A galactic microquasar mimicking winged radio galaxiesA galactic microquasar mimicking winged radio galaxies
A galactic microquasar mimicking winged radio galaxiesSérgio Sacani
 
Ento-322, Agrochemicals for agriculture usee
Ento-322, Agrochemicals for agriculture useeEnto-322, Agrochemicals for agriculture usee
Ento-322, Agrochemicals for agriculture useeDrAnita Sharma
 
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...AmalDhivaharS
 
green chemistry, clean sustainable environment.ppt
green chemistry, clean sustainable environment.pptgreen chemistry, clean sustainable environment.ppt
green chemistry, clean sustainable environment.pptRashmiSanghi1
 
Exploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptx
Exploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptxExploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptx
Exploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptxSamrat Tayade
 
Open Access Publishing and the Open Journal of Astrophysics
Open Access Publishing and the Open Journal of AstrophysicsOpen Access Publishing and the Open Journal of Astrophysics
Open Access Publishing and the Open Journal of AstrophysicsPeter Coles
 
Introduction to the research of stem cells
Introduction to the research of stem cellsIntroduction to the research of stem cells
Introduction to the research of stem cellsAlaaOraby6
 
Chemistry chapter 1 solutions detailed explanation
Chemistry chapter 1 solutions detailed explanationChemistry chapter 1 solutions detailed explanation
Chemistry chapter 1 solutions detailed explanationayuqroyjohn85
 
Study of X - Ray Spectra and its types
Study  of X  - Ray Spectra and its typesStudy  of X  - Ray Spectra and its types
Study of X - Ray Spectra and its typestanishashukla147
 
The ExoGRAVITY project - observations of exoplanets from the ground with opti...
The ExoGRAVITY project - observations of exoplanets from the ground with opti...The ExoGRAVITY project - observations of exoplanets from the ground with opti...
The ExoGRAVITY project - observations of exoplanets from the ground with opti...Advanced-Concepts-Team
 
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...Abhinav S
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsPeter Coles
 
2024 Insilicogen Company English Brochure
2024 Insilicogen Company English Brochure2024 Insilicogen Company English Brochure
2024 Insilicogen Company English BrochureInsilico Gen
 
Rootstock scion and Interstock Relationship Selection of Elite Mother Plants
Rootstock scion and Interstock Relationship Selection of Elite Mother PlantsRootstock scion and Interstock Relationship Selection of Elite Mother Plants
Rootstock scion and Interstock Relationship Selection of Elite Mother PlantsAmanDohre
 
Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.Naresh Gupta
 
conceptofatomic#number Physical Science second sem week 1.pptx
conceptofatomic#number Physical Science second sem week 1.pptxconceptofatomic#number Physical Science second sem week 1.pptx
conceptofatomic#number Physical Science second sem week 1.pptxAnggeComeso
 
PROSTHETIC FEET description and its types
PROSTHETIC FEET description and its typesPROSTHETIC FEET description and its types
PROSTHETIC FEET description and its typeseshasmalik27
 
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022Nan Yang Academy of Sciences
 

Recently uploaded (20)

INTRODUCTION TO PLANT TAXONOMY WITH DIVERSE TAXONOMIC APPROACHES
INTRODUCTION TO PLANT TAXONOMY WITH DIVERSE TAXONOMIC APPROACHESINTRODUCTION TO PLANT TAXONOMY WITH DIVERSE TAXONOMIC APPROACHES
INTRODUCTION TO PLANT TAXONOMY WITH DIVERSE TAXONOMIC APPROACHES
 
commercial production of cellulase enzyme and its uses
commercial production of cellulase enzyme and its usescommercial production of cellulase enzyme and its uses
commercial production of cellulase enzyme and its uses
 
A galactic microquasar mimicking winged radio galaxies
A galactic microquasar mimicking winged radio galaxiesA galactic microquasar mimicking winged radio galaxies
A galactic microquasar mimicking winged radio galaxies
 
Ento-322, Agrochemicals for agriculture usee
Ento-322, Agrochemicals for agriculture useeEnto-322, Agrochemicals for agriculture usee
Ento-322, Agrochemicals for agriculture usee
 
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
 
green chemistry, clean sustainable environment.ppt
green chemistry, clean sustainable environment.pptgreen chemistry, clean sustainable environment.ppt
green chemistry, clean sustainable environment.ppt
 
Exploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptx
Exploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptxExploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptx
Exploring Artificial Intelligence_ Revolutionizing Tomorrow's World.pptx
 
Open Access Publishing and the Open Journal of Astrophysics
Open Access Publishing and the Open Journal of AstrophysicsOpen Access Publishing and the Open Journal of Astrophysics
Open Access Publishing and the Open Journal of Astrophysics
 
Introduction to the research of stem cells
Introduction to the research of stem cellsIntroduction to the research of stem cells
Introduction to the research of stem cells
 
Chemistry chapter 1 solutions detailed explanation
Chemistry chapter 1 solutions detailed explanationChemistry chapter 1 solutions detailed explanation
Chemistry chapter 1 solutions detailed explanation
 
Study of X - Ray Spectra and its types
Study  of X  - Ray Spectra and its typesStudy  of X  - Ray Spectra and its types
Study of X - Ray Spectra and its types
 
The ExoGRAVITY project - observations of exoplanets from the ground with opti...
The ExoGRAVITY project - observations of exoplanets from the ground with opti...The ExoGRAVITY project - observations of exoplanets from the ground with opti...
The ExoGRAVITY project - observations of exoplanets from the ground with opti...
 
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
 
2024 Insilicogen Company English Brochure
2024 Insilicogen Company English Brochure2024 Insilicogen Company English Brochure
2024 Insilicogen Company English Brochure
 
Rootstock scion and Interstock Relationship Selection of Elite Mother Plants
Rootstock scion and Interstock Relationship Selection of Elite Mother PlantsRootstock scion and Interstock Relationship Selection of Elite Mother Plants
Rootstock scion and Interstock Relationship Selection of Elite Mother Plants
 
Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.
 
conceptofatomic#number Physical Science second sem week 1.pptx
conceptofatomic#number Physical Science second sem week 1.pptxconceptofatomic#number Physical Science second sem week 1.pptx
conceptofatomic#number Physical Science second sem week 1.pptx
 
PROSTHETIC FEET description and its types
PROSTHETIC FEET description and its typesPROSTHETIC FEET description and its types
PROSTHETIC FEET description and its types
 
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
 

Featured

PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Applitools
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at WorkGetSmarter
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...DevGAMM Conference
 

Featured (20)

Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
 

2021-11, EMNLP, Distilling Relation Embeddings from Pre-trained Language Models

  • 1. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio Jose Camacho-Collados Steven Schockaert https://github.com/asahi417/relbert
  • 2. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Language Model Understanding 2 Syntactic Knowledge - Probing embedding: Hewitt 2019, Tenney 2019 - Probing attention weight: Clark 2020 Factual Knowledge a.k.a Language Model as a Commonsense KB - Petroni 2019, Kassner 2020, Jiang 2020, etc Relational Knowledge a.k.a Language Model as a Lexical Relation Reasoner - LM fine-tuning on relation classification: Bouraoui 2019 - Vanilla LM evaluation: Ushio 2021
  • 3. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Language Model Understanding 3 Syntactic Knowledge - Probing embedding: Hewitt 2019, Tenney 2019 - Probing attention weight: Clark 2020 Factual Knowledge a.k.a Language Model as a Commonsense KB - Petroni 2019, Kassner 2020, Jiang 2020, etc Relational Knowledge a.k.a Language Model as a Lexical Relation Reasoner - LM fine-tuning on relation classification: Bouraoui 2019 - Vanilla LM evaluation: Ushio 2021
  • 4. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Language Model Understanding 4 Syntactic Knowledge - Probing embedding: Hewitt 2019, Tenney 2019 - Probing attention weight: Clark 2020 Factual Knowledge a.k.a Language Model as a Commonsense KB - Petroni 2019, Kassner 2020, Jiang 2020, etc Relational Knowledge a.k.a Language Model as a Lexical Relation Reasoner - LM fine-tuning on relation classification: Bouraoui 2019 - Vanilla LM evaluation: Ushio 2021 Can we distil relational knowledge as relation embedding?
  • 5. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Relation Embedding Word Embedding Mikolov (2013) Pair2Vec Joshi (2019) Relative Camacho-Collados (2019) 5 King Queen Woman Man Word Embedding
  • 7. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Prompt Generation Custom Template, AutoPrompt (Shin 2020), P-tuning (Liu 2021) LM embedding Averaging over the context Relation Embedding from LM 7 (h, t) Word pair Sentence (tokens) Prompt Generation Contextual embeddings Fixed-length embedding LM encoding x Averaging s1 s2 s3
  • 8. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Prompt Generation Custom Template, AutoPrompt (Shin 2020), P-tuning (Liu 2021) LM embedding Averaging over the context Relation Embedding from LM 8 (h, t) Word pair Sentence (tokens) Prompt Generation Contextual embeddings Fixed-length embedding LM encoding x Averaging s1 s2 s3 1. Today, I finally discovered the relation between camera and photographer : camera is the <mask> of photographer 2. Today, I finally discovered the relation between camera and photographer : photographer is camera’s <mask> 3. Today, I finally discovered the relation between camera and photographer : <mask> 4. I wasn’t aware of this relationship, but I just read in the encyclopedia that camera is the <mask> of photographer 5. I wasn’t aware of this relationship, but I just read in the encyclopedia that photographer is camera’s <mask>
  • 9. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Given a triple: anchor “sunscreen::sunburn”, positive “vaccine::malaria”, and negative “dog::cat”, we want the embeddings of the anchor and the posirive close but far from the negative. Loss function: Triplet loss and classification loss following SBERT (Reimers 2019). Fine-tuning on Triples 9 vaccine::malaria sunscreen::sunburn dog::cat
  • 10. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados We create the dataset from SemEval 2012 Task 2. Dataset 10 Class Inclusion Parent Relation Child Relation Taxonomic Functional (h1,t1), ... , (hk,tk) Positive (h1,t1), ... , (hk,tk) Negative : Triplet ( xa, xp, xn ) : : ( xa,1, xp,1, xn,1 ) ( xa,2, xp,2, xn,2 ) : ( xa,m, xp,m, xn,m ) [(xa,m, xp,m, xa,1), ..., (xa,m, xp,m, xp,m-1)] Original batch: m samples Augmented batch: 2m(m-1) samples [(xa,1, xp,1, xa,2 ), ..., (xa,1, xp,1, xp,m)]
  • 12. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Experiment: Analogy 12 Setup - Cosine similarity in between embeddings. - No training. - Accuracy as the metric. - No validation. Sample from SAT analogy dataset. Data statistics.
  • 13. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Experiment: Analogy 13 SotA in 4 / 5 datasets 🎉 Better than tuned methods on dev set 🤗
  • 14. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Experiment: Classification 14 Setup - Supervised Task - LMs are frozen - macro/micro F1 - Tuned on dev Data statistics.
  • 15. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados SotA in 4 / 5 datasets in macro F1 score 🎉 SotA in 3 / 5 datasets in micro F1 score 🎉 Experiment: Classification 15
  • 17. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Relation Memorarization 17 Does RelBERT just memorize the relations in the training set… ? Experiment: Train RelBERT without hypernymy. Result: No significant decrease in hyperbyny prediction. → RelBERT does not rely on the memorization!
  • 18. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Fine-tuning? Other LMs? 18 Train RelBERT on BERT, ALBERT in addition to RoBERTa. → RoBERTa is the best. Vanilla RoBERTa (no fine-tuning). → Fine-tuning (distillation) is necessary.
  • 19. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Conclusion 19 ● We propose RelBERT, a framework to achieve relation embedding model based on pretrained LM. ● RelBERT distil the LM’s relational knowledge and realize a high quality relation embedding. ● Experimental results show that RelBERT embedding outperform existing baselines, establishing SotA in analogy and relation classification.
  • 20. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Release of RelBERT Library 20 We release python package relbert (install via pip install relbert) along with model checkpoints on the huggingface modelhub. Please check our project page https://github.com/asahi417/relbert !! from relbert import RelBERT model = RelBERT( 'asahi417/relbert-roberta-large' ) # the vector has (1024,) v_tokyo_japan = model.get_embedding([ 'Tokyo', 'Japan'])
  • 22. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Comparing to Word Embeddings 22 FastText is still better than RelBERT in Google Analogy Question. Breakdown per relation types shows that FastText is better in the morphological relation, while very poor in the lexical relation.
  • 23. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Nearest Neighbours 23
  • 24. Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio, Steven Schockaert, and Jose Camacho-Collados Fine-tuning on Triples Given a triple of the anchor (eg. “sunscreen”), the positive (eg. “sunburn”), and the negative (eg. “evil”), the triplet loss is defined as and the classification loss is defined as where is a learnable weight. The loss functions are inspired by SBERT (Reimers 2019). 24