AAAI-19 paper: Neural Relation ExtractionWithin and Across Sentence Boundaries Authors: Pankaj Gupta, Subburam Rajaram, Hinrich Schuetze and Thomas Runkler

1. Intern © Siemens AG 2017
Neural Relation Extraction Within and Across
Sentence Boundaries
Author(s): Pankaj Gupta1,2, Subburam Rajaram2, Hinrich Schütze1 and Thomas Runkler2
Presenter: Pankaj Gupta @AAAI19 Honolulu Hawaii, USA. Jan 2019
1CIS, University of Munich (LMU) | 2 Machine Intelligence, Siemens AG | Jan 2019

2. Intern © Siemens AG 2017
May 2017Seite 2 Corporate Technology
Outline
Introduction: Relation Extraction spanning sentence boundaries
Proposed Methods
➢ Inter-sentential Dependency-Based Neural Networks (iDepNN)
→ Inter-sentential Shortest Dependency Path (iDepNN-SDP)
→ Inter-sentential Augmented Dependency Path (iDepNN-ADP)
Evaluation and Analysis
➢ State-of-the-art comparison
➢ Error analysis

3. Intern © Siemens AG 2017
May 2017Seite 3 Corporate Technology
Outline
Introduction: Relation Extraction spanning sentence boundaries
Proposed Methods
➢ Inter-sentential Dependency-Based Neural Networks (iDepNN)
→ Inter-sentential Shortest Dependency Path (iDepNN-SDP)
→ Inter-sentential Augmented Dependency Path (iDepNN-ADP)
Evaluation and Analysis
➢ State-of-the-art comparison
➢ Error analysis

4. Intern © Siemens AG 2017
May 2017Seite 4 Corporate Technology
Outline
Introduction: Relation Extraction spanning sentence boundaries
Proposed Methods
➢ Inter-sentential Dependency-Based Neural Networks (iDepNN)
→ Inter-sentential Shortest Dependency Path (iDepNN-SDP)
→ Inter-sentential Augmented Dependency Path (iDepNN-ADP)
Evaluation and Analysis
➢ State-of-the-art comparison
➢ Error analysis

5. Intern © Siemens AG 2017
May 2017Seite 5 Corporate Technology
Outline
Introduction: Relation Extraction spanning sentence boundaries
Proposed Methods
➢ Inter-sentential Dependency-Based Neural Networks (iDepNN)
→ Inter-sentential Shortest Dependency Path (iDepNN-SDP)
→ Inter-sentential Augmented Dependency Path (iDepNN-ADP)
Evaluation and Analysis
➢ State-of-the-art comparison
➢ Error analysis

6. Intern © Siemens AG 2017
May 2017Seite 6 Corporate Technology
Introduction: Relation Extraction (RE)
Binary Relation Extraction(RE):
- identify semantic relationship between a pair of nominals or entities e1 and e2 in a given text snippet, S

7. Intern © Siemens AG 2017
May 2017Seite 7 Corporate Technology
Introduction: Relation Extraction (RE)
Binary Relation Extraction(RE):
- identify semantic relationship between a pair of nominals or entities e1 and e2 in a given text snippet, S
Paul Allen has started a company and named [Vern Raburn]e1 its [president]e2 .
relation: per-post(e1,e2)

8. Intern © Siemens AG 2017
May 2017Seite 8 Corporate Technology
Introduction: Relation Extraction (RE)
Relation Extraction
(Based on location of entities)
Binary Relation Extraction(RE):
- identify semantic relationship between a pair of nominals or entities e1 and e2 in a given text snippet, S

9. Intern © Siemens AG 2017
May 2017Seite 9 Corporate Technology
Introduction: Relation Extraction (RE)
intra-sentential
(entities within sentence boundary)
most
prior works
Binary Relation Extraction(RE):
- identify semantic relationship between a pair of nominals or entities e1 and e2 in a given text snippet, S
Relation Extraction
(Based on location of entities)

10. Intern © Siemens AG 2017
May 2017Seite 10 Corporate Technology
Introduction: Relation Extraction (RE)
intra-sentential
(entities within sentence boundary)
most
prior works
This work
Binary Relation Extraction(RE):
- identify semantic relationship between a pair of nominals or entities e1 and e2 in a given text snippet, S
Relation Extraction
(Based on location of entities)
inter-sentential
(entities across sentence boundary(s))

11. Intern © Siemens AG 2017
May 2017Seite 11 Corporate Technology
Introduction: Relation Extraction (RE)
intra-sentential
(entities within sentence boundary)
most
prior works
This work
Binary Relation Extraction(RE):
- identify semantic relationship between a pair of nominals or entities e1 and e2 in a given text snippet, S
Relation Extraction
(Based on location of entities)
Paul Allen has started a company and named [Vern Raburn]e1 its [president]e2 .
relation: per-post(e1,e2)
Example
inter-sentential
(entities across sentence boundary(s))

12. Intern © Siemens AG 2017
May 2017Seite 12 Corporate Technology
Introduction: Relation Extraction (RE)
intra-sentential
(entities within sentence boundary)
most
prior works
This work
Binary Relation Extraction(RE):
- identify semantic relationship between a pair of nominals or entities e1 and e2 in a given text snippet, S
Relation Extraction
(Based on location of entities)
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company, to be called [Paul Allen Group] e2 will be based in Bellevue, Washington.
relation: per-org(e1,e2)
inter-sentential
(entities across sentence boundary(s))
Example

13. Intern © Siemens AG 2017
May 2017Seite 13 Corporate Technology
Introduction: Relation Extraction (RE)
intra-sentential
(entities within sentence boundary)
most
prior works
This work
Binary Relation Extraction(RE):
- identify semantic relationship between a pair of nominals or entities e1 and e2 in a given text snippet, S
Relation Extraction
(Based on location of entities)
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company, to be called [Paul Allen Group] e2 will be based in Bellevue, Washington.
relation: ??
inter-sentential
(entities across sentence boundary(s))
MISSED relationships:
Impact the system
performance, leading
to POOR RECALL

14. Intern © Siemens AG 2017
May 2017Seite 14 Corporate Technology
Introduction: Relation Extraction (RE)
intra-sentential
(entities within sentence boundary)
most
prior works
This work
Binary Relation Extraction(RE):
- identify semantic relationship between a pair of nominals or entities e1 and e2 in a given text snippet, S
Relation Extraction
(Based on location of entities)
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company, to be called [Paul Allen Group] e2 will be based in Bellevue, Washington.
relation: per-org(e1,e2)
inter-sentential
(entities across sentence boundary(s))
Capture relationship
between entities at
distance across
sentence boundaries

15. Intern © Siemens AG 2017
May 2017Seite 15 Corporate Technology
Challenges in Inter-sentential Relation Extraction (RE)
This work
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company will coordinate the overall strategy for the group of high-tech
companies that Mr. Allen owns or holds a significant stake in, will be based in
Bellevue, Washington and called [Paul Allen Group] e2 .
relation: per-org(e1,e2)
inter-sentential
(entities across sentence boundary(s))

16. Intern © Siemens AG 2017
May 2017Seite 16 Corporate Technology
Challenges in Inter-sentential Relation Extraction (RE)
This work
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company will coordinate the overall strategy for the group of high-tech
companies that Mr. Allen owns or holds a significant stake in, will be based in
Bellevue, Washington and called [Paul Allen Group] e2 .
relation: per-org(e1,e2)
inter-sentential
(entities across sentence boundary(s))
NOISY text in relationships
spanning sentence boundaries:
POOR PRECISION

17. Intern © Siemens AG 2017
May 2017Seite 17 Corporate Technology
Challenges in Inter-sentential Relation Extraction (RE)
This work
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company will coordinate the overall strategy for the group of high-tech
companies that Mr. Allen owns or holds a significant stake in, will be based in
Bellevue, Washington and called [Paul Allen Group] e2 .
relation: per-org(e1,e2)
inter-sentential
(entities across sentence boundary(s))
NOISY text in relationships
spanning sentence boundaries:
POOR PRECISION
Robust system to
tackle false positives
in inter-sentential RE
Need

18. Intern © Siemens AG 2017
May 2017Seite 18 Corporate Technology
This Work: Contribution
Propose a novel neural approach for Inter-sentential Relation Extraction

19. Intern © Siemens AG 2017
May 2017Seite 19 Corporate Technology
This Work: Contribution
1. Neural architecture based on dependency parse trees
➔ named as inter-sentential Dependency-based Neural Network (iDepNN)
2. Unified neural framework of a bidirectional RNN (biRNN) and Recursive NN (RecNN)
3. Extract relations within and across sentence boundaries by modeling:
➔ shortest dependency path (SDP) using biRNN
➔ augmented dependency path (ADP) using RecRNN
Propose a novel neural approach for Inter-sentential Relation Extraction
Contribution

20. Intern © Siemens AG 2017
May 2017Seite 20 Corporate Technology
This Work: Contribution
1. Neural architecture based on dependency parse trees
➔ named as inter-sentential Dependency-based Neural Network (iDepNN)
2. Unified neural framework of a bidirectional RNN (biRNN) and Recursive NN (RecNN)
3. Extract relations within and across sentence boundaries by modeling:
➔ shortest dependency path (SDP) using biRNN
➔ augmented dependency path (ADP) using RecRNN
Propose a novel neural approach for Inter-sentential Relation Extraction
Contribution

21. Intern © Siemens AG 2017
May 2017Seite 21 Corporate Technology
This Work: Contribution
1. Neural architecture based on dependency parse trees
➔ named as inter-sentential Dependency-based Neural Network (iDepNN)
2. Unified neural framework of a bidirectional RNN (biRNN) and Recursive NN (RecNN)
3. Extract relations within and across sentence boundaries by modeling:
➔ shortest dependency path (SDP) using biRNN
➔ augmented dependency path (ADP) using RecRNN
Propose a novel neural approach for Inter-sentential Relation Extraction
Contribution

22. Intern © Siemens AG 2017
May 2017Seite 22 Corporate Technology
This Work: Contribution
1. Neural architecture based on dependency parse trees
➔ named as inter-sentential Dependency-based Neural Network (iDepNN)
2. Unified neural framework of a bidirectional RNN (biRNN) and Recursive NN (RecNN)
3. Extract relations within and across sentence boundaries by modeling:
➔ shortest dependency path (SDP) using biRNN
➔ augmented dependency path (ADP) using RecRNN
Propose a novel neural approach for Inter-sentential Relation Extraction
Contribution
1. precisely extract relationships within and across sentence boundaries
2. show a better balance in precision and recall with an improved F1 score
Benefits

23. Intern © Siemens AG 2017
May 2017Seite 23 Corporate Technology
Dependency Based Relation Extraction
Sentences and their dependency graphs
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company, to be called [Paul Allen Group] e2 will be based in Bellevue, Washington.
Shortest Dependency Path (SDP)
between root to entity e1

24. Intern © Siemens AG 2017
May 2017Seite 24 Corporate Technology
Dependency Based Relation Extraction
Sentences and their dependency graphs
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company, to be called [Paul Allen Group] e2 will be based in Bellevue, Washington.
Shortest Dependency Path (SDP)
between root to entity e2

25. Intern © Siemens AG 2017
May 2017Seite 25 Corporate Technology
Dependency Based Relation Extraction
Sentences and their dependency graphs Inter-sentential Shortest Dependency Path
(iSDP) across sentence boundary.
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company, to be called [Paul Allen Group] e2 will be based in Bellevue, Washington.
iSDP
→ Connection between the roots of adjacent sentences by NEXTS

26. Intern © Siemens AG 2017
May 2017Seite 26 Corporate Technology
Dependency Based Relation Extraction
Sentences and their dependency graphs Inter-sentential Shortest Dependency Path
(iSDP) across sentence boundary.
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company, to be called [Paul Allen Group] e2 will be based in Bellevue, Washington.
subtreeiSDP
→ Connection between the roots of adjacent sentences by NEXTS

27. Intern © Siemens AG 2017
May 2017Seite 27 Corporate Technology
Dependency Based Relation Extraction
Sentences and their dependency graphs Inter-sentential Shortest Dependency Path
(iSDP) across sentence boundary.
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company, to be called [Paul Allen Group] e2 will be based in Bellevue, Washington.
subtreeiSDP
→ Connection between the roots of adjacent sentences by NEXTS

28. Intern © Siemens AG 2017
May 2017Seite 28 Corporate Technology
Dependency Based Relation Extraction
Sentences and their dependency graphs Inter-sentential Shortest Dependency Path
(iSDP) across sentence boundary.
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company, to be called [Paul Allen Group] e2 will be based in Bellevue, Washington.
subtreeiSDP
→ Connection between the roots of adjacent sentences by NEXTS

29. Intern © Siemens AG 2017
May 2017Seite 29 Corporate Technology
Dependency Based Relation Extraction
Sentences and their dependency graphs Inter-sentential Shortest Dependency Path
(iSDP) across sentence boundary.
Paul Allen has started a company and named [Vern Raburn]e1 its president. The
company, to be called [Paul Allen Group] e2 will be based in Bellevue, Washington.
subtreeiSDP
→ Connection between the roots of adjacent sentences by NEXTS

30. Intern © Siemens AG 2017
May 2017Seite 30 Corporate Technology
Proposed Approach: Intra- and inter-sentential RE

31. Intern © Siemens AG 2017
May 2017Seite 31 Corporate Technology
Proposed Approach: Intra- and inter-sentential RE
Inter-sentential Dependency-based Neural Network variants: iDepNN-SDP and iDepNN-ADP

32. Intern © Siemens AG 2017
May 2017Seite 32 Corporate Technology
Proposed Approach: Intra- and inter-sentential RE
Inter-sentential Dependency-based Neural Network variants: iDepNN-SDP

33. Intern © Siemens AG 2017
May 2017Seite 33 Corporate Technology
Proposed Approach: Intra- and inter-sentential RE
Inter-sentential Dependency-based Neural Network variants: iDepNN-SDP

34. Intern © Siemens AG 2017
May 2017Seite 34 Corporate Technology
Proposed Approach: Intra- and inter-sentential RE
Inter-sentential Dependency-based Neural Network variants: iDepNN-ADP
subtree

35. Intern © Siemens AG 2017
May 2017Seite 35 Corporate Technology
Proposed Approach: Intra- and inter-sentential RE
Inter-sentential Dependency-based Neural Network variants: iDepNN-ADP
Compute subtree embedding
subtree

36. Intern © Siemens AG 2017
May 2017Seite 36 Corporate Technology
Proposed Approach: Intra- and inter-sentential RE
Inter-sentential Dependency-based Neural Network variants: iDepNN-ADP
Compute subtree embedding
subtree
→ Offers precise
structure
→ Offers additional
information in
classifying
relation

37. Intern © Siemens AG 2017
May 2017Seite 37 Corporate Technology
Evaluation and Analysis

38. Intern © Siemens AG 2017
May 2017Seite 38 Corporate Technology
Evaluation and Analysis: Datasets
Datasets
➢ evaluate on four datasets from medical and news domain
Count of intra- and inter-sentential relationships in datasets

39. Intern © Siemens AG 2017
May 2017Seite 39 Corporate Technology
Evaluation and Analysis: Datasets
Datasets
➢ evaluate on four datasets from medical and news domain
Count of intra- and inter-sentential relationships in datasets

40. Intern © Siemens AG 2017
May 2017Seite 40 Corporate Technology
Evaluation and Analysis: Datasets
Datasets
➢ evaluate on four datasets from medical and news domain
Count of intra- and inter-sentential relationships in datasets

41. Intern © Siemens AG 2017
May 2017Seite 41 Corporate Technology
Evaluation and Analysis: Datasets
Datasets
➢ evaluate on four datasets from medical and news domain
Count of intra- and inter-sentential relationships in datasets

42. Intern © Siemens AG 2017
May 2017Seite 42 Corporate Technology
Evaluation and Analysis: Datasets
Datasets
➢ evaluate on four datasets from medical and news domain
Count of intra- and inter-sentential relationships in datasets

43. Intern © Siemens AG 2017
May 2017Seite 43 Corporate Technology
Evaluation and Analysis: Datasets
Datasets
➢ evaluate on four datasets from medical and news domain
Count of intra- and inter-sentential relationships in datasets
Result
discussed in
this talk

44. Intern © Siemens AG 2017
May 2017Seite 44 Corporate Technology
Evaluation and Analysis: Datasets
Datasets
➢ evaluate on four datasets from medical and news domain
Baselines:
→ SVM, graphLSTM, i-biRNN and i-biLSTM
Count of intra- and inter-sentential relationships in datasets
Result
discussed in
this talk

45. Intern © Siemens AG 2017
May 2017Seite 45 Corporate Technology
Results (Precision / Recall / F1)

46. Intern © Siemens AG 2017
May 2017Seite 46 Corporate Technology
Results (Precision / Recall / F1): Intra-sentential training
iDepNN-ADP is precise in inter-sentential RE
than both SVM and graphLSTM
precise

47. Intern © Siemens AG 2017
May 2017Seite 47 Corporate Technology
Results (Precision / Recall / F1): Intra-sentential training
iDepNN-ADP outperforms both SVM and graphLSTM
in terms of F1 in inter-sentential RE due to a better
balance in precision and recall
F1

48. Intern © Siemens AG 2017
May 2017Seite 48 Corporate Technology
Results (Precision / Recall / F1): Inter-sentential training
iDepNN-ADP outperforms both SVM and graphLSTM
in terms of P and F1 in inter-sentential RE
F1

49. Intern © Siemens AG 2017
May 2017Seite 49 Corporate Technology
Results (Precision / Recall / F1): Inter-sentential training
F1

50. Intern © Siemens AG 2017
May 2017Seite 50 Corporate Technology
Results (Precision / Recall / F1): Ensemble

51. Intern © Siemens AG 2017
May 2017Seite 51 Corporate Technology
Ensemble with Thresholding on Prediction Probability
Ensemble scores at various thresholds
p: output probability
pr: the count of predictions.

52. Intern © Siemens AG 2017
May 2017Seite 52 Corporate Technology
Official Scores: State-of-the-art
Ensemble scores at various thresholds
p: output probability
pr: the count of predictions. Official results on test set: Comparison with the
published systems in the BioNLP ST 2016.

53. Intern © Siemens AG 2017
May 2017Seite 53 Corporate Technology
Error Analysis: BioNLP ST 2016 dataset

54. Intern © Siemens AG 2017
May 2017Seite 54 Corporate Technology
Error Analysis: BioNLP ST 2016 dataset

55. Intern © Siemens AG 2017
May 2017Seite 55 Corporate Technology
Error Analysis: BioNLP ST 2016 dataset
Few false positives
in iDepNN-ADP,
compared to both
SVM and graphLSTM
iDepNN-ADP
SVM
graphLSTM

56. Intern © Siemens AG 2017
May 2017Seite 56 Corporate Technology
Key Takeaways
➢ Propose a novel neural approach iDepNN for Inter-sentential Relation Extraction
➢ Precisely extract relations within and across sentence boundaries by modeling:
➔ shortest dependency path (SDP) using biRNN, i.e., iDepNN-SDP
➔ augmented dependency path (ADP) using RecRNN, i.e., iDepNN-ADP
➢ Demonstrate a better balance in precision and recall with an improved F1 score
➢ Evaluate on 4 datasets from news and medical domains
➢ Achieve a gain of 5.2% (0.587 vs 0.558) in F1 over the winning team (out of 11 teams)
in BioNLP Shared Task (ST) 2016
Code and Data: https://github.com/pgcool/Cross-sentence-Relation-Extraction-iDepNN

57. Intern © Siemens AG 2017
May 2017Seite 57 Corporate Technology
Thanks !!!
Code and Data: https://github.com/pgcool/Cross-sentence-Relation-Extraction-iDepNN