250414_JH_labseminar[SPECTER: Document-level Representation Learnin...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

SPECTER : Document-level
Representation Learning using
Citation-informed Transformers
Ju-Hee SHIM
Network Science Lab
Dept...

2
 INTRODUCTION
• Motivation
 Architecture
• SPECTER
• Method
 Evaluation
• Datasets
• State-of-the-art methods
• Exper...

3
INTRODUCTION
Motivation
 Problems of Existing Transformer-based Language Models
• Rapid growth of scientific literature...

4
Architecture
 To learn high-quality document-level representation:
• Using citations as an inter-document relatedness s...

5
Architecture
Method
 Triplet sampling:
• Each training instance is a triplet of papers :
• Query paper :
• Positive pap...

6
Architecture
Method
 Encoder:
• Use SciBERT for foundation and fine-tune it with Semantic Scholar data :
• Encoder Inpu...

7
Architecture
Method
 SciDocs (Benchmark dataset):
1. Document Classification (F1-Score)
• Medical Subject Headings Clas...

8
Datasets
Evaluation
 Semantic Scholar corpus
 146K query papers
 Around 26.7M tokens
 32K papers for validation
 5 ...

9
Evaluation
State-of-the-art methods
 Compared Methods
• SIF
• SciBERT
• Sentence-BERT
• Citeomatic
• SGC

10
Evaluation
Experimental setup
•Evaluation Metrics
• Ranking
• NDCG
• MAP
•Optimizer: Adam
•Learning Rate: 2e-5
•Schedul...

11
Evaluation
Results
 Overall, significant improvements on almost tasks except for SGC which has an access to citation g...

12
Evaluation
Results
 Ablation Study Results
• Removing Abstract degrades performance
• Adding meta-data such as Author ...

13
Evaluation
Results
 Ablation Study Results
• SPECTER works significantly better than Fine-Tuned SciBERT.

14
Conclusion
Conclusion
 The author of this paper was proposing SPECTER, a novel approach to apply inter-document relate...

15
Q & A
Q / A

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1010457 views
Storytelling For The Web: Integrate... by Chiara Aliotta 918112 views
Artificial Intelligence, Data and C... by OECD Directorate ... 818086 views
How to Leverage AI to Boost Employe... by SocialHRCamp 341372 views
2024 State of Marketing Report – by... by Marius Sescu 194920 views
Everything You Need To Know About C... by Expeed Software 218230 views

View on Slideshare

1 of 15 Ad

View on Slideshare

1 of 15 Ad