IMPACT Final Conference - Language Parallel Sessions - Landsbergen

2,373 views

Published on

Published in: Education, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,373
On SlideShare
0
From Embeds
0
Number of Embeds
1,943
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

IMPACT Final Conference - Language Parallel Sessions - Landsbergen

  1. 1. Named Entity Work in IMPACT Dr. Frank Landsbergen, Institute for Dutch Lexicology
  2. 2. Structure of this talk <ul><li>Named Entities (NE‘s): definition </li></ul><ul><li>NE‘s in OCR and end-user retrieval </li></ul><ul><li>NE-lexicon building </li></ul>
  3. 3. Structure of this talk <ul><li>Named Entities (NE‘s): definition </li></ul><ul><li>NE‘s in OCR and end-user retrieval </li></ul><ul><li>NE-lexicon building </li></ul>
  4. 4. What are named entities? <ul><li>The Examiner (September 1845) </li></ul>
  5. 5. What are named entities? <ul><li>The Examiner (September 1845) </li></ul><ul><li>Named entity: a word or string of words referring to a proper location, person, or organisation (or date, time, etc.) </li></ul>
  6. 6. <ul><li>Named Entities (NE‘s): definition </li></ul><ul><li>NE‘s in OCR and end-user retrieval </li></ul><ul><li>NE-lexicon building </li></ul>
  7. 7. From image to OCR’ed text On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of the palace of the Emperor Frederick Barbarossa, and Hanau, a tolerably large place containing about 14,000 inhabitants, and the scene of more than one warlike exploit. It was gallantly defended for nine months during the thirty years' war, and in later days Napoleon, while in hot retreat from Leipsic, dashed through the allied troops of the Austrians Bavarians under its walls. The Queen reached I i'ankfort at three o'clock in the afternoon, and descended at the Hotel d'Angleterre, where a body of the City guard was drawn up to receive her Majesty. In the evening, after dinner, there was a reception or levee, at which the Larl and Countess of Westmoreland, the Burgomaster, and some others of the authorities of Frankfort were pre- sented. Prince Metternich and the King of Bavaria dined with the Royal party.
  8. 8. Retrieval On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of the palace of the Emperor Frederick Barbarossa, and Hanau, a tolerably large place containing about 14,000 inhabitants, and the scene of more than one warlike exploit. It was gallantly defended for nine months during the thirty years' war, and in later days Napoleon, while in hot retreat from Leipsic, dashed through the allied troops of the Austrians Bavarians under its walls. The Queen reached I i'ankfort at three o'clock in the afternoon, and descended at the Hotel d'Angleterre, where a body of the City guard was drawn up to receive her Majesty. In the evening, after dinner, there was a reception or levee, at which the Larl and Countess of Westmoreland, the Burgomaster, and some others of the authorities of Frankfort were pre- sented. Prince Metternich and the King of Bavaria dined with the Royal party. User searches for: Napoleon
  9. 9. Retrieval On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of the palace of the Emperor Frederick Barbarossa, and Hanau, a tolerably large place containing about 14,000 inhabitants, and the scene of more than one warlike exploit. It was gallantly defended for nine months during the thirty years' war, and in later days Napoleon , while in hot retreat from Leipsic, dashed through the allied troops of the Austrians Bavarians under its walls. The Queen reached I i'ankfort at three o'clock in the afternoon, and descended at the Hotel d'Angleterre, where a body of the City guard was drawn up to receive her Majesty. In the evening, after dinner, there was a reception or levee, at which the Larl and Countess of Westmoreland, the Burgomaster, and some others of the authorities of Frankfort were pre- sented. Prince Metternich and the King of Bavaria dined with the Royal party. User searches for: Napoleon
  10. 10. Retrieval On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of the palace of the Emperor Frederick Barbarossa, and Hanau, a tolerably large place containing about 14,000 inhabitants, and the scene of more than one warlike exploit. It was gallantly defended for nine months during the thirty years' war, and in later days Napoleon, while in hot retreat from Leipsic, dashed through the allied troops of the Austrians Bavarians under its walls. The Queen reached I i'ankfort at three o'clock in the afternoon, and descended at the Hotel d'Angleterre, where a body of the City guard was drawn up to receive her Majesty. In the evening, after dinner, there was a reception or levee, at which the Larl and Countess of Westmoreland, the Burgomaster, and some others of the authorities of Frankfort were pre- sented. Prince Metternich and the King of Bavaria dined with the Royal party. User searches for: Frederic?k? Barbarosa
  11. 11. Retrieval On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of the palace of the Emperor Frederick Barbarossa , and Hanau, a tolerably large place containing about 14,000 inhabitants, and the scene of more than one warlike exploit. It was gallantly defended for nine months during the thirty years' war, and in later days Napoleon, while in hot retreat from Leipsic, dashed through the allied troops of the Austrians Bavarians under its walls. The Queen reached I i'ankfort at three o'clock in the afternoon, and descended at the Hotel d'Angleterre, where a body of the City guard was drawn up to receive her Majesty. In the evening, after dinner, there was a reception or levee, at which the Larl and Countess of Westmoreland, the Burgomaster, and some others of the authorities of Frankfort were pre- sented. Prince Metternich and the King of Bavaria dined with the Royal party. User searches for: Frederic?k? Barbarosa
  12. 12. Retrieval On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of the palace of the Emperor Frederick Barbarossa , and Hanau, a tolerably large place containing about 14,000 inhabitants, and the scene of more than one warlike exploit. It was gallantly defended for nine months during the thirty years' war, and in later days Napoleon, while in hot retreat from Leipsic, dashed through the allied troops of the Austrians Bavarians under its walls. The Queen reached I i'ankfort at three o'clock in the afternoon, and descended at the Hotel d'Angleterre, where a body of the City guard was drawn up to receive her Majesty. In the evening, after dinner, there was a reception or levee, at which the Larl and Countess of Westmoreland, the Burgomaster, and some others of the authorities of Frankfort were pre- sented. Prince Metternich and the King of Bavaria dined with the Royal party. User searches for: Frederic?k? Barbarosa
  13. 13. Retrieval On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of the palace of the Emperor Frederick Barbarossa , and Hanau, a tolerably large place containing about 14,000 inhabitants, and the scene of more than one warlike exploit. It was gallantly defended for nine months during the thirty years' war, and in later days Napoleon, while in hot retreat from Leipsic , dashed through the allied troops of the Austrians Bavarians under its walls. The Queen reached I i'ankfort at three o'clock in the afternoon, and descended at the Hotel d'Angleterre, where a body of the City guard was drawn up to receive her Majesty. In the evening, after dinner, there was a reception or levee, at which the Larl and Countess of Westmoreland, the Burgomaster, and some others of the authorities of Frankfort were pre- sented. Prince Metternich and the King of Bavaria dined with the Royal party. User searches for: Leipzig
  14. 14. Retrieval On Thursday, the 4th inst., her Majesty continued her journey through Gelnhausen, where stand the ruins of the palace of the Emperor Frederick Barbarossa , and Hanau, a tolerably large place containing about 14,000 inhabitants, and the scene of more than one warlike exploit. It was gallantly defended for nine months during the thirty years' war, and in later days Napoleon, while in hot retreat from Leipsic, dashed through the allied troops of the Austrians Bavarians under its walls. The Queen reached I i'ankfort at three o'clock in the afternoon, and descended at the Hotel d'Angleterre, where a body of the City guard was drawn up to receive her Majesty. In the evening, after dinner, there was a reception or levee, at which the Larl and Countess of Westmoreland, the Burgomaster, and some others of the authorities of Frankfort were pre- sented. Prince Metternich and the King of Bavaria dined with the Royal party. User searches for: Frankfurt
  15. 15. Retrieval <ul><li>Enable user to find NE’s regardless of historical spelling variation </li></ul><ul><li>Leipsic, Dares Salem, Frankfort </li></ul><ul><li>Improve quality of OCR for NE’s </li></ul><ul><li>I i'ankfort, Larl of Westmoreland </li></ul>
  16. 16. <ul><li>Named Entities (NE‘s): definition </li></ul><ul><li>NE‘s in OCR and end-user retrieval </li></ul><ul><li>NE-lexicon building </li></ul>
  17. 17. Building an NE-lexicon <ul><li>Database of named entities </li></ul><ul><li>Related entries (wordforms) in the database are linked </li></ul><ul><li>Entries are enriched (e.g. PND) </li></ul>Amsterdam Amsterdams Amsteldam Aemsteldam Amstelredamme Amsterdam
  18. 18. Building an NE-lexicon <ul><li>Database of named entities </li></ul><ul><li>Related entries (wordforms) in the database are linked </li></ul><ul><li>Entries are enriched (e.g. PND) </li></ul>Frederik Hendrik PND-id: 118590200 Frederik Frederick Frederic Fredric Fr édéric Frederiks Amsterdam Amsterdams Amsteldam Aemsteldam Amstelredamme Amsterdam
  19. 19. Building an NE-lexicon: a 4 step-process <ul><li>data collection (NE-lists, text) </li></ul><ul><li>NE-tagging </li></ul><ul><li>enrichment (POS-tagging, lemmatizing, adding name structure, variants) </li></ul><ul><li>database creation </li></ul>
  20. 20. Building an NE-lexicon: NE-tagging <ul><li>NE-extraction software </li></ul><ul><li>Stanford University, with added module </li></ul><ul><li>Statistical NE-recognition: </li></ul><ul><ul><li>Software ‘trains itself’ with manually tagged data </li></ul></ul><ul><ul><li>automatic search for features + weights that ‘predict’ NE’s in tagged text </li></ul></ul><ul><ul><li>uses trained model to mark NE’s in untagged text </li></ul></ul><ul><li>Manual tagging of training data: Attestation Tool </li></ul>
  21. 21. NE-recognition <ul><li>Statistical NE-recognition software works very well with modern data. But how about historical material? </li></ul><ul><li>Benchmark tests, comparing results for modern & historical data </li></ul>
  22. 22. NE-recognition <ul><li>NE-recognition on historical data not necessarily poorer than on modern data </li></ul><ul><li>More important is the resemblance between training set and extraction set (time period, genre) </li></ul><ul><li>Added spelling variation module improves performance by ‘smoothening’ variation </li></ul>
  23. 23. Building an NE-lexicon: enrichment <ul><li>Lemmatization </li></ul><ul><li>POS-tagging </li></ul><ul><li>Person name structure </li></ul><ul><li>Variant matching </li></ul><ul><li>Link to autority files </li></ul><ul><li>NE-variant matching module that links: </li></ul><ul><li>Historical wordform to modern lemma </li></ul><ul><li>Aemsteldam > Amsterdam </li></ul><ul><li>Historical wordforms to each other </li></ul><ul><li>Frederik, Fredrik, Fréderick </li></ul>
  24. 24. The NE-lexicon
  25. 25. IMPACT-work on NE‘s so far <ul><li>Toolkit for NE-lexicon building available (NERT, Attestation Tool) </li></ul><ul><li>NE-lexica of Dutch, English, German </li></ul>
  26. 26. IMPACT-work on NE‘s so far <ul><li>Toolkit for NE-lexicon building available (NERT, Attestation Tool) </li></ul><ul><li>NE-lexica of Dutch, English, German </li></ul>Hendricus Hamer Hendricus Hamerus Hendrikkus Hindrikus Hendericus Henderikus Heudricus Henricus Hendrikus Henriçus henriecus Hamer Amerus Haamer Hammer
  27. 27. Contact <ul><li>Frank Landsbergen </li></ul><ul><li>Institute for Dutch Lexicology </li></ul><ul><li>Leiden, Netherlands </li></ul><ul><li>f [email_address] </li></ul>

×