This document proposes a method to train dependency parser models without manually annotated data by: 1) Creating an initial "blank" model that outputs unlabeled dependencies 2) Parsing an unannotated corpus with the blank model and logging the decisions 3) Extracting heuristics from the parse decisions to identify common dependency patterns 4) Generating new training examples by modifying the logged decisions based on the heuristics 5) Retraining the model on the generated examples, which improves the parser outputs over the blank model. Further work to optimize efficiency and heuristic analysis is suggested.

1. CORPORA-BASED GENERATION OF
DEPENDENCY PARSER MODELS FOR
NATURAL LANGUAGE PROCESSING
by Edmond Lepedus
supervised by Marek Grześ, Christian Kissig and Laura Bocchi

2. BACKGROUND

3. Dependency Parsing

4. Dependency Parsing
• Structure consists of dependencies between
words:

5. Dependency Parsing
• Structure consists of dependencies between
words:
Hello world

6. Dependency Parsing
• Structure consists of dependencies between
words:
Hello world
dependent

7. Dependency Parsing
• Structure consists of dependencies between
words:
Hello world
headdependent

8. Stanford CoreNLP

9. Stanford CoreNLP
• Free, open-source NLP toolkit

10. Stanford CoreNLP
• Free, open-source NLP toolkit
• Includes a dependency parser backed by a neural
network classifier

11. Stanford CoreNLP
• Free, open-source NLP toolkit
• Includes a dependency parser backed by a neural
network classifier
• Parses 1000 sentences per second at 92.2% accuracy:

12. Stanford CoreNLP
• Free, open-source NLP toolkit
• Includes a dependency parser backed by a neural
network classifier
• Parses 1000 sentences per second at 92.2% accuracy:

13. Stanford CoreNLP
• Free, open-source NLP toolkit
• Includes a dependency parser backed by a neural
network classifier
• Parses 1000 sentences per second at 92.2% accuracy:
• Trained on manually annotated text

14. AIM

15. Aim
Train a classifier using an unparsed corpus of
English language text

16. MOTIVATION

17. Motivation

18. Motivation
• Decrease the cost of training data

19. Motivation
• Decrease the cost of training data
• Increase the availability of training data

20. Motivation
• Decrease the cost of training data
• Increase the availability of training data
• Increase parsing accuracy

21. Motivation
• Decrease the cost of training data
• Increase the availability of training data
• Increase parsing accuracy
• Enable the parsing of languages with few
remaining speakers

22. APPROACH

23. Overview

24. Overview
1. Create a ‘blank’ model

25. Overview
1. Create a ‘blank’ model
2. Parse corpus with model & log decisions

26. Overview
1. Create a ‘blank’ model
2. Parse corpus with model & log decisions
3. Extract heuristics from corpus & parse log

27. Overview
1. Create a ‘blank’ model
2. Parse corpus with model & log decisions
3. Extract heuristics from corpus & parse log
4. Generate training examples by modifying the
logged decisions to fit the discovered heuristics

28. Overview
1. Create a ‘blank’ model
2. Parse corpus with model & log decisions
3. Extract heuristics from corpus & parse log
4. Generate training examples by modifying the
logged decisions to fit the discovered heuristics
5. Train model on new examples

29. Diagram
Create a ‘blank’ model
Parse & log
Extract heuristics
Generate training examples
Train new model

30. IMPLEMENTATION

31. Blank Model Creation

32. Blank Model Creation
• Outputs left arcs with custom ‘unknown’ label:

33. Blank Model Creation
• Outputs left arcs with custom ‘unknown’ label:

34. Blank Model Creation
• Outputs left arcs with custom ‘unknown’ label:
• Supports the creation of new training examples

35. Parse Decision Logs

36. Parse Decision Logs
• Log every parse decision to YAML:

37. Parse Decision Logs
• Log every parse decision to YAML:

38. Parse Decision Logs
• Log every parse decision to YAML:
required for training

39. Heuristic Extraction

40. Heuristic Extraction
• Count bigram occurrences:

41. Heuristic Extraction
• Count bigram occurrences:

42. Heuristic Extraction
• Count bigram occurrences:
• Assume that frequent bigrams indicate
dependency

43. Training Example Generation

44. Training Example Generation

45. Training Example Generation

46. Training Example Generation

47. Training Example Generation

48. RESULTS

49. Post-training Parses

50. Post-training Parses

51. Post-training Parses

52. FURTHER WORK

53. Further Work

54. Further Work
• Improve efficiency to enable the use of larger
corpora

55. Further Work
• Improve efficiency to enable the use of larger
corpora
• Develop better heuristic analyses

56. Further Work
• Improve efficiency to enable the use of larger
corpora
• Develop better heuristic analyses
• Implement arc labels

57. CONCLUSION

58. Conclusion

59. Conclusion
• We modified the Stanford CoreNLP toolkit to
enable the creation of ‘blank’ parser models

60. Conclusion
• We modified the Stanford CoreNLP toolkit to
enable the creation of ‘blank’ parser models
• We developed a workflow for training parser
models without using annotated corpora

61. Conclusion
• We modified the Stanford CoreNLP toolkit to
enable the creation of ‘blank’ parser models
• We developed a workflow for training parser
models without using annotated corpora
• We showed that this quickly yields qualitative
improvements in parser outputs over the ‘blank’
models

62. Conclusion
• We modified the Stanford CoreNLP toolkit to
enable the creation of ‘blank’ parser models
• We developed a workflow for training parser
models without using annotated corpora
• We showed that this quickly yields qualitative
improvements in parser outputs over the ‘blank’
models
• We proposed three avenues for further research

63. ANY QUESTIONS?

64. REFERENCES
[1] C. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. Bethard, and D. McClosky,
“The Stanford CoreNLP Natural Language Processing Toolkit,” presented at the
Proceedings of 52nd Annual Meeting of the Association for Computational
Linguistics: System Demonstrations, Stroudsburg, PA, USA, 2014, pp. 55–60.
[2] J. Nivre, “Dependency Parsing,” vol. 4, no. 3, pp. 138–152, Mar. 2010.
[3] D. Chen and C. D. Manning, “A Fast and Accurate Dependency Parser
using Neural Networks.,” EMNLP, pp. 740–750, 2014.
[4] D. Jurafsky and J. H. Martin, Speech and language processing: an
introduction to natural language processing, computational linguistics, and
speech recognition. Upper Saddle River, NJ: Prentice-Hall, 2000.