Named Entities

1. NAMED ENTITIES March 2016 1

2. NAMED ENTITIES 2 PLAN What? More what? How? How well? Where to? Why?

3. NAMED ENTITIES 3

4. NAMED ENTITIES 4 Kabosu

5. NAMED ENTITIES 5 DEFINITIONS The names of i.e.: • persons • organisations • locations • expressions of times • quantities • monetary values • percentages

6. NAMED ENTITY RECOGNITION 6 INTERACTIVE A Maine bill would allow residents of the state’s island communities to ship medical samples by ferry rather than in person. Democrats in the State Senate say the proposal was prompted by changes in the Maine State Ferry Service that make it difficult to ship samples, such as bloodwork, from islands. New policies say Maine ferries won’t transport lab work for patients anymore. That means island residents must travel to mainland hospitals to deliver the samples, which can take hours. Sen. Dave Miramant, a Camden Democrat, says a “lock box” for samples should be available on all boats. The bill was the subject of a Jan. 28 public hearing where some North Haven island residents spoke in favor of it. The state Legislature’s transportation committee will review the bill soon.

7. NAMED ENTITY RECOGNITION 7 STANFORD ONLINE NER A Maine bill would allow residents of the state’s island communities to ship medical samples by ferry rather than in person. Democrats in the State Senate say the proposal was prompted by changes in the Maine State Ferry Service that make it difficult to ship samples, such as bloodwork, from islands. New policies say Maine ferries won’t transport lab work for patients anymore. That means island residents must travel to mainland hospitals to deliver the samples, which can take hours. Sen. Dave Miramant, a Camden Democrat, says a “lock box” for samples should be available on all boats. The bill was the subject of a Jan. 28 public hearing where some North Haven island residents spoke in favor of it. The state Legislature’s transportation committee will review the bill soon.

8. NAMED ENTITY RECOGNITION 8 STANFORD ONLINE NER

9. NAMED ENTITY RECOGNITION 9 DEFINITIONS • Find the names of entities in text • State-of-the-art NER systems for English produce near-human performance. For example, the best system entering MUC-7 scored 93.39% of F-measure while human annotators scored 97.60% and 96.95%

10. NAMED ENTITY RECOGNITION 10 APPROACHES • Detection • Classification into entity type • Approaches • Grammar (rule) based • Statistical • Machine Learned

11. NAMED ENTITY RECOGNITION 11 APPROACHES • Current state-of-the-art: • Conditional Random Field

12. NAMED ENTITY RECOGNITION 12 FUTURE • Deep Learning • Based on Word2Vec • Enhance with sequence information (memory)

13. NAMED ENTITY RECOGNITION 13 OUR MOTIVATION

14. 14 Purpose Help companies grow their business faster by finding the most relevant prospects Enable companies to get strategic sales information within seconds Approach A big data company using intelligent algorithms to turn business data into insights and services Combine company data, key people profiles and relevant news in a unique, real-time platform Companies 163,000,000 companies world-wide People 160,000,000 key executives News Over 2,000,000 news articles per day Worldwide 212 countries covered Triggers 1,400,000 news triggers per month FACTSHEET Structured Structured Feed Generated

15. NAMED ENTITY RECOGNITION 15 OUR MOTIVATION Unstructured data • Crawl company data • Currently only crawling English companies • ~1B web pages

17. NAMED ENTITY RECOGNITION 17 OUR MOTIVATION Extraction: • Find people in crawled data • Find relevant business information in crawled data • Same for news

18. NAMED ENTITY RECOGNITION 18 OUR MOTIVATION Matching: • Unstructured data supporting structured data • News articles are matched to the right company • Triggers in news

22. NAMED ENTITIES 22 RECOGNITION EXPERIMENTS

23. NAMED ENTITY RECOGNITION 23 STANFORD NER • http://nlp.stanford.edu/software/CRF-NER.shtml • Dual license including GPL v2 • Conditional Random Field sequence model

24. NAMED ENTITY RECOGNITION 24 STANFORD NER • Detects many entities • Detects companies, person names, titles, locations • Detects many things that are not entities • Easily fooled by Titlecase • Easily fooled by abbreviations

25. NAMED ENTITY RECOGNITION 25 INTERACTIVE A Maine bill would allow residents of the state’s island communities to ship medical samples by ferry rather than in person. Democrats in the State Senate say the proposal was prompted by changes in the Maine State Ferry Service that make it difficult to ship samples, such as bloodwork, from islands. New policies say Maine ferries won’t transport lab work for patients anymore. That means island residents must travel to mainland hospitals to deliver the samples, which can take hours. Sen. Dave Miramant, a Camden Democrat, says a “lock box” for samples should be available on all boats. The bill was the subject of a Jan. 28 public hearing where some North Haven island residents spoke in favor of it. The state Legislature’s transportation committee will review the bill soon.

26. NAMED ENTITY RECOGNITION 26 STANFORD NER Democrats in the State Senate say the proposal was prompted by changes in the Maine State Ferry Service that make it difficult to ship samples, …, from islands. … The state Legislature’s transportation committee will review the bill soon. Identified as organisations: • State Senate • Maine State Ferry Service • Legislature

27. NAMED ENTITY RECOGNITION 27 STANFORD NER Democrats in the State Senate say the proposal was prompted by changes in the Maine State Ferry Service that make it difficult to ship samples, …, from islands. … The state Legislature’s transportation committee will review the bill soon. Not really named - should be: • State Senate - The Main State Senate • Legislature - The Main State Legislature

28. NAMED ENTITY RECOGNITION 28 STANFORD NER Sen. Dave Miramant, a Camden Democrat, says Identified as Person: • Dave Miramant

29. NAMED ENTITY RECOGNITION 29 STANFORD NER A Maine bill would allow residents of the state’s island communities to ship medical samples by ferry rather than in person. Identified as Location: • Maine

30. NAMED ENTITY RECOGNITION 30 STANFORD NER

32. NAMED ENTITY RECOGNITION 32 OUR MOTIVATION Matching: • The correct company • The correct person • The correct location DISAMBIGUATION!

33. NAMED ENTITY DISAMBIGUATION 33 THE PROBLEM

34. NAMED ENTITY DISAMBIGUATION 34 THE PROBLEM • Which Apple, Apple Inc. or Apple Corps

35. NAMED ENTITY DISAMBIGUATION 35 THE PROBLEM • Agfa Apogee or Apogee Electronics?

36. NAMED ENTITIES 36 DISAMBIGUATION • ERD Challenge at SIGIR 2014 • Most (all) solutions based around Web Search (Bing) and Wikipedia

37. NAMED ENTITIES 37 DISAMBIGUATION SMAPH

38. NAMED ENTITY DISAMBIGUTAION 38 SMAPH • Annotator - source 1 • Normal Search - source 2 • Wikisearch - source 3

39. NAMED ENTITY DISAMBIGUTAION 39 SMAPH 1. Fetching – from a search engine 2. Spotting – parse results to identify candidate mentions for the entities to be annotated. 3. Candidate generation • from the Wikipedia pages occurring in the search results • from an existing annotator, using the mentions identified in the spotting step 4. Pruning – binary SVM classifier

41. NAMED ENTITY DISAMBIGUTAION 41 SMAPH F1: 62.9%

42. NAMED ENTITY DISAMBIGUTAION 42 SMAPH • 60+% F1 score for disambiguation is good • The 90+% F1-score was for recognition

43. NAMED ENTITIES 43 DISAMBIGUATION EXPERIMENTS

44. WIKIPEDIA 44 Numbers Pacific Standard none Ohio State Ohio_State_Buckeyes Panama Golf Club none Poors Reafirms Ecopetrol S.A. none Ecopetrol S.A. BOGOTA Ecopetrol Standard and Poors Standard_%26_Poor%27s Ecopetrol Ecopetrol Poors none Company none WVEC WVEC Sentara Norfolk General Hospital Sentara_Norfolk_General_Hospital Google Google

45. WIKIPEDIA 45 GOLD SET NUMBERS • Total number of detected entities: 328 • Actual positive: 197 • Actual negative: 131 Should have been a job for Mechanical Turk Not all detected entities are actually entities

46. WIKIPEDIA 46 SYSTEM • Docker • Elastic search v2.2 • English wikipedia index • wikiparse: https://github.com/andrewvc/wikiparse

47. WIKIPEDIA 47 SCORING • Actual Negative - none • Actual Positive - something other than none • Predicted positive - something, not necessarily the correct one • Predicted negative - none Predicted positive and incorrect is treated as actual negative

48. WIKIPEDIA 48 QUERY POST { "query": { "bool": { "disable_coord": "True", "should": [ { "term": { "title": "nyse" } }, { "term": { "body": "nyse" } }, { "term": { "body": "zacks investment research" } }, { "term": { "body": "pg&e corporation" } }, { "term": { "body": "pg&e co." } } ], "must": { "term": { "title": "nyse" } }, "minimum_should_match": 1 } } }

49. WIKIPEDIA 49 INITIAL RESULTS Actual Positive Actual Negative Predicted Positive 23 52 Predicted Negative 138 115 31% 14%

50. NAMED ENTITY DISAMBIGUTAION 50 SCORE F1: 0.19

51. WIKIPEDIA 51 COMMON FALSE POSITIVE SITUATION • LLC - Wikipedia, the free encyclopedia • https://en.wikipedia.org/wiki/LLC • LLC may refer to: Air transport[edit]. LLC, LHD Landing Craft, Australian variant of the LCM-1E landing craft; LLC, ICAO airline designator of FlyLAL Charters ...

52. WIKIPEDIA 52 SCORE THRESHOLD Actual Positive Actual Negative Predicted Positive 22 26 Predicted Negative 153 127 46% 13%

54. WIKIPEDIA 54 NO MANDATORY TITLE MATCH { "query": { "bool": { "disable_coord": "True", "should": [ { "term": { "title": "nyse" } }, { "term": { "body": "nyse" } }, { "term": { "body": "zacks investment research" } }, { "term": { "body": "pg&e corporation" } }, { "term": { "body": "pg&e co." } } ], "must": { "term": { "title": "nyse" } }, "minimum_should_match": 1 } } }

55. WIKIPEDIA 55 NO MANDATORY TITLE MATCH Actual Positive Actual Negative Predicted Positive 23 290 Predicted Negative 8 7 7% 74%

57. WIKIPEDIA 57 COMMON FALSE POSITIVE SITUATION { "query": { "bool": { "disable_coord": "True", "should": [ { "term": { "title": "nyse" } }, { "term": { "body": "nyse" } }, { "term": { "body": "zacks investment research" } }, { "term": { "body": "pg&e corporation" } }, { "term": { "body": "pg&e co." } } ], "must": { "term": { "title": "nyse" } }, "minimum_should_match": 1 } } }

58. WIKIPEDIA 58 NO OPTIONAL TITLE MATCH Actual Positive Actual Negative Predicted Positive 1 312 Predicted Negative 8 7

59. NAMED ENTITY DISAMBIGUTAION 59 MY BEST RESULT F1: 19%

60. LOW SCORE 60 EXPLANATION • No web search used • No annotator • Only a wikipedia search

62. OTHER SOURCES 62 WEB SEARCH

63. WEB SEARCH 63 WHY NOT? • Google prohibits bot requests • Bing allows paid API queries - expensive • Building our own web index is too expensive

64. ALTERNATIVES 64 DATABASE OF COMPANIES • If we had a database of companies • Find the company name aliases • Find key people • Products?

65. BUSINESS SEARCH ENGINE 65 COMPANY SEARCH

69. BUSINESS SEARCH ENGINE 69 ENTITY SEARCH

70. BUSINESS SEARCH ENGINE 70 ENTITY SEARCH

72. NAMED ENTITY 72 Knut O. Hellan • CTO Companybook • Twitter: @KnutHellan

Named Entities

You are reading a preview.

Check these out next

Named Entities

More Related Content

Viewers also liked (9)

Similar to Named Entities (9)

Recently uploaded (20)

Named Entities

Named Entities

You are reading a preview.

Named Entities

Recommended

More Related Content

Viewers also liked (9)

Similar to Named Entities (9)

Recently uploaded (20)

Named Entities