SlideShare a Scribd company logo
1 of 38
Corefrence Resolution A Machine Learning Approach nicolas_ nicolov @ jdpa .com shumin. wu @ colorado .edu Shumin Wu Ph.D. Candidate in Computer Science University of Colorado at Boulder The Center for Spoken Language Research  1777 Exposition Drive Boulder, Colorado 80301, U.S.A. Nicolas Nicolov Senior Director, Science J.D. Power and Associates, McGraw-Hill Web Intelligence Division 4888 Pearl East Circle Boulder, CO 80301, U.S.A.
[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Outline
Coreference Audi   is an automaker that makes luxury cars and SUVs.  The company   was born in  Germany  .  It   was established by  August Horch   in 1910.  Horch   had previosly founded another company and  his   models were quite popular.  Audi   started with four cylinder models. By 1914,  Horch  's new cars were racing and winning.  August Horch   left  the Audi company   in 1920 to take a position as an industry representative for the  German   motor vehicle industry federation. Currently  Audi   is a subsidiary of the  Volkswagen group   and produces cars of outstanding quality.
[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Coreference Applications
MUC6 F-measure a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Count the number of corresponding links between mentions Precision = 4/5 Recall = 4/6 F-measure = 2* Precision * Recall/( Precision + Recall ) = 0.727
MUC6 F-measure Degenerate Case a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Precision =  N/A Recall = 0 F-measure =  N/A a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Discounts single mention entities All mentions  form individual singleton entities.
MUC6 F-measure Degenerate Case 2 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Precision = 6/8 Recall = 1 F-measure =  0.857 !!! a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Does not adequately penalize dense links. All mentions form one big entity.
Roadmap ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
B 3  F-measure a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: For each mention, compute the proportion of corresponding mentions between reference and system entity. Precision =1/9*( Recall = 1/9*( F-measure = 0.760 3/3 +3/3 +3/3 +1/3 +2/3 +2/3 +1/1 +2/2 +2/2 ) = 0.852 3/4 +3/4 +3/4 +1/4 +2/2 +2/2 +1/3 +2/3 +2/3 ) = 0.685
B 3  F-measure Degenerate Case a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Precision = 1 Recall = 1/3 F-measure = 0.5 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 All mentions  form individual singleton entities.
B 3  F-measure Degenerate Case 2 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Precision = 1/9*(4/9+4/9+4/9+4/9+2/9+2/9+3/9+3/9+3/9) = 0.358 Recall = 1 F-measure = 0.527 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Which system entity maps to which reference entity? All mentions form one big entity.
Roadmap ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
CEAF a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Find the one-to-one entity mapping between reference (R) and system (S) maximizing  similarity measure
CEAF Degenerate Case a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 All mentions  form individual singleton entities.
CEAF Degenerate Case 2 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 All mentions form one big entity.
Performance Measures Summary ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Roadmap ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
ICWSM JDPA Coreference Corpus ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
ICWSM JDPA Corpus: Mention Types ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
ICWSM JDPA Corpus Statistics ,[object Object],[object Object],mentions entities with x mentions in them *Entity x DocId. “Audi” in doc5 and in doc7 are considered different entities. Mention type Entities* Person 13,505 Organization 8,206 Location 245 City 360 US State 127 Country 316 Nationality 195 Facility 1,052 Vehicles 13,977 CarPart 15,478 CarFeature 6,119 Camera 5,197 CameraFeature 9,575 CameraPart 3,240 CameraAccessory 352 Descriptor 13,087 … …
Roadmap ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Coreference Approaches ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Mention Pair Features Considered ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Roadmap ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Coreference Heuristics ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Heuristic System: Results Order of clustering (local (mentions within sentence) to global, forward, and reverse direction) did not alter our results. Configuration MUC-F MUC-P MUC-R B 3 -F B 3 -P B 3 -R Unlinked entities -- -- 0 71.8 100 56.0 Single entity (w/ all mentions) 61.4 44.3 100 8.6 4.5 100 w/ editing distant 66.2 64.0 68.5 78.6 76.8 80.4 editing dist. at sentence level 64.8 63.7 66.0 78.7 77.8 79.4 w/o editing distant 70.0 75.8 64.9 83.0 86.8 79.5 Edit dist + cardinality match 70.4 78.7 63.7 83.7 89.2 78.8
Roadmap ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Machine Learning Approach ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Robust Risk Minimization (RRM) RRM was proposed by Tong Zhang; best CoNLL’03 chunker. Separate positive and negative weights Multiplicative weight update
Bell Tree Input sequence: a 1 , b 1 , b 2 , a 2 … Lots of states in the search space to explore! [ a 1 ]  [a 1 , b 1 ] [a 1 ][ b 1 ] [a 1 ,b 1 , b 2 ] [a 1 ,b 1 ][ b 2 ] [a 1 , b 2 ][b 1 ] [a 1 ][b 1 , b 2 ] [a 1 ][b 1 ][ b 2 ] [a 1 ,b 1 ,b 2 , a 2 ] [a 1 ,b 1 ,b 2 ][ a 2 ] [a 1 , a 2 ] [b 1 ,b 2 ] [a 1 ][b 1 ,b 2 , a 2 ] [a 1 ][b 1 ,b 2 ][ a 2 ] [a 1 ,b 1 ][b 2 ][ a 2 ] [a 1 ,b 1 ,  a 2 ][b 2 ] [a 1 ,b 1 ][b 2 , a 2 ] [a 1 ,b 2 , a 2 ][b 1 ] [a 1 ,b 2 ][b 1 , a 2 ] [a 1 ,b 2 ][b 1 ][ a 2 ] [a 1 , a 2 ][b 1 ][b 2 ] [a 1 ][b 1 , a 2 ][b 2 ] [a 1 ][b 1 ][b 2 , a 2 ] [a 1 ][b 1 ][b 2 ] [ a 2 ]
Bell Tree Coreference Model ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Entities assumed to be independent  Link probability derived from the most probable mention pair Tunable to encourage or penalize entity creation
Bell Tree in Action Input sequence: a 1 , b 1 , b 2 , a 2 , c 1 … Coreference probability: [ a 1 ]  p = 1 [a 1 , b 1 ] p=0.4 a 1 b 1 b 2 a 2 c 1 a 1 1 b 1 0.4 1 b 2 0.2 0.9 1 a 2 0.8 0.1 0.3 1 c 1 0.4 0.3 0.4 0.2 1 [a 1 ][ b 1 ] p=0.6 [a 1 ,b 1 , b 2 ] p=0.36 [a 1 ,b 1 ][ b 2 ] p=0.04 [a 1 , b 2 ][b 1 ] p=0.12 [a 1 ][b 1 , b 2 ] p=0.54 [a 1 ][b 1 ][ b 2 ] p=0.06 [a 1 ,b 1 ,b 2 , a 2 ] p=0.288 [a 1 ,b 1 ,b 2 ][ a 2 ] p=0.072 [a 1 , a 2 ] [b 1 ,b 2 ] p=0.432 [a 1 ][b 1 ,b 2 , a 2 ] p=0.162 [a 1 ][b 1 ,b 2 ][ a 2 ] p=0.108 [a 1 ,b 1 ,b 2 , a 2 , c 1 ] p=0.1152 [a 1 ,b 1 ,b 2 , a 2 ][ c 1 ] p=0.1728 [a 1 ,a 2, c 1 ][b 1 , b 2 ] p=0.1728 [a 1 ,a 2 ][b 1 , b 2, c 1 ] p=0.1728 [a 1 ,a 2 ][b 1 , b 2 ][ c 1 ] p=0.2592
Discussion ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Future Work ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Acknowledgements ,[object Object],[object Object],[object Object],[object Object],[object Object]
References ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Thank you!

More Related Content

Similar to Coreference Resolution

Software EngineeringBackground for Question 1-7 Kean Universi.docx
Software EngineeringBackground for Question 1-7 Kean Universi.docxSoftware EngineeringBackground for Question 1-7 Kean Universi.docx
Software EngineeringBackground for Question 1-7 Kean Universi.docxwhitneyleman54422
 
ROSeAnn Presentation
ROSeAnn PresentationROSeAnn Presentation
ROSeAnn PresentationDBOnto
 
Assignment 2 specificationThe assignment must be submitted e.docx
Assignment 2 specificationThe assignment must be submitted e.docxAssignment 2 specificationThe assignment must be submitted e.docx
Assignment 2 specificationThe assignment must be submitted e.docxsteviesellars
 
RELIABILITY OF MECHANICAL SYSTEM OF SYSTEMS
RELIABILITY OF MECHANICAL SYSTEM OF SYSTEMSRELIABILITY OF MECHANICAL SYSTEM OF SYSTEMS
RELIABILITY OF MECHANICAL SYSTEM OF SYSTEMScscpconf
 
Workshop on requirements and modeling at HAE 2015
Workshop on requirements and modeling at HAE 2015Workshop on requirements and modeling at HAE 2015
Workshop on requirements and modeling at HAE 2015Olivier Béghain
 
Matrix Mapper in Aviation Industry
Matrix Mapper in Aviation IndustryMatrix Mapper in Aviation Industry
Matrix Mapper in Aviation Industryaarkentechnologies
 
Airline flights delay prediction- 2014 Spring Data Mining Project
Airline flights delay prediction- 2014 Spring Data Mining ProjectAirline flights delay prediction- 2014 Spring Data Mining Project
Airline flights delay prediction- 2014 Spring Data Mining ProjectHaozhe Wang
 
Week 06Conjoint Analysishttpswww.smh.com.au.docx
Week 06Conjoint Analysishttpswww.smh.com.au.docxWeek 06Conjoint Analysishttpswww.smh.com.au.docx
Week 06Conjoint Analysishttpswww.smh.com.au.docxjessiehampson
 
IMPLEMENTATION OF MACHINE LEARNING IN E-COMMERCE & BEYOND
IMPLEMENTATION OF MACHINE LEARNING IN E-COMMERCE & BEYONDIMPLEMENTATION OF MACHINE LEARNING IN E-COMMERCE & BEYOND
IMPLEMENTATION OF MACHINE LEARNING IN E-COMMERCE & BEYONDRabi Das
 
DOMAIN ENGINEERING FOR APPLIED MONOCULAR RECONSTRUCTION OF PARAMETRIC FACES
DOMAIN ENGINEERING FOR APPLIED MONOCULAR RECONSTRUCTION OF PARAMETRIC FACESDOMAIN ENGINEERING FOR APPLIED MONOCULAR RECONSTRUCTION OF PARAMETRIC FACES
DOMAIN ENGINEERING FOR APPLIED MONOCULAR RECONSTRUCTION OF PARAMETRIC FACESsipij
 
Domain Engineering for Applied Monocular Reconstruction of Parametric Faces
Domain Engineering for Applied Monocular Reconstruction of Parametric FacesDomain Engineering for Applied Monocular Reconstruction of Parametric Faces
Domain Engineering for Applied Monocular Reconstruction of Parametric Facessipij
 
2005 Web Content Mining 4
2005 Web Content Mining   42005 Web Content Mining   4
2005 Web Content Mining 4George Ang
 
Sadcw 7e chapter03-done(1)
Sadcw 7e chapter03-done(1)Sadcw 7e chapter03-done(1)
Sadcw 7e chapter03-done(1)LamineKaba6
 
IRJET - Airplane Crash Analysis and Prediction using Machine Learning
IRJET - Airplane Crash Analysis and Prediction using Machine LearningIRJET - Airplane Crash Analysis and Prediction using Machine Learning
IRJET - Airplane Crash Analysis and Prediction using Machine LearningIRJET Journal
 
Decision Making in Product Design
Decision Making in Product DesignDecision Making in Product Design
Decision Making in Product DesignZaid Haque
 
American Psychological Association (APA) information for .docx
American Psychological Association (APA) information for .docxAmerican Psychological Association (APA) information for .docx
American Psychological Association (APA) information for .docxdaniahendric
 
Your Parts Data and Why You NEED SPEEDcat!
Your Parts Data and Why You NEED SPEEDcat!Your Parts Data and Why You NEED SPEEDcat!
Your Parts Data and Why You NEED SPEEDcat!Gigi Ho
 
Basics (Reports Writing 2)
Basics (Reports Writing 2)Basics (Reports Writing 2)
Basics (Reports Writing 2)julianmillar
 
autonomous_vehicle_working_paper_01072020-_508_compliant.pdf
autonomous_vehicle_working_paper_01072020-_508_compliant.pdfautonomous_vehicle_working_paper_01072020-_508_compliant.pdf
autonomous_vehicle_working_paper_01072020-_508_compliant.pdfPandurangGurakhe
 

Similar to Coreference Resolution (20)

Software EngineeringBackground for Question 1-7 Kean Universi.docx
Software EngineeringBackground for Question 1-7 Kean Universi.docxSoftware EngineeringBackground for Question 1-7 Kean Universi.docx
Software EngineeringBackground for Question 1-7 Kean Universi.docx
 
ROSeAnn Presentation
ROSeAnn PresentationROSeAnn Presentation
ROSeAnn Presentation
 
Assignment 2 specificationThe assignment must be submitted e.docx
Assignment 2 specificationThe assignment must be submitted e.docxAssignment 2 specificationThe assignment must be submitted e.docx
Assignment 2 specificationThe assignment must be submitted e.docx
 
RELIABILITY OF MECHANICAL SYSTEM OF SYSTEMS
RELIABILITY OF MECHANICAL SYSTEM OF SYSTEMSRELIABILITY OF MECHANICAL SYSTEM OF SYSTEMS
RELIABILITY OF MECHANICAL SYSTEM OF SYSTEMS
 
Workshop on requirements and modeling at HAE 2015
Workshop on requirements and modeling at HAE 2015Workshop on requirements and modeling at HAE 2015
Workshop on requirements and modeling at HAE 2015
 
Matrix Mapper in Aviation Industry
Matrix Mapper in Aviation IndustryMatrix Mapper in Aviation Industry
Matrix Mapper in Aviation Industry
 
Airline flights delay prediction- 2014 Spring Data Mining Project
Airline flights delay prediction- 2014 Spring Data Mining ProjectAirline flights delay prediction- 2014 Spring Data Mining Project
Airline flights delay prediction- 2014 Spring Data Mining Project
 
Week 06Conjoint Analysishttpswww.smh.com.au.docx
Week 06Conjoint Analysishttpswww.smh.com.au.docxWeek 06Conjoint Analysishttpswww.smh.com.au.docx
Week 06Conjoint Analysishttpswww.smh.com.au.docx
 
IMPLEMENTATION OF MACHINE LEARNING IN E-COMMERCE & BEYOND
IMPLEMENTATION OF MACHINE LEARNING IN E-COMMERCE & BEYONDIMPLEMENTATION OF MACHINE LEARNING IN E-COMMERCE & BEYOND
IMPLEMENTATION OF MACHINE LEARNING IN E-COMMERCE & BEYOND
 
DOMAIN ENGINEERING FOR APPLIED MONOCULAR RECONSTRUCTION OF PARAMETRIC FACES
DOMAIN ENGINEERING FOR APPLIED MONOCULAR RECONSTRUCTION OF PARAMETRIC FACESDOMAIN ENGINEERING FOR APPLIED MONOCULAR RECONSTRUCTION OF PARAMETRIC FACES
DOMAIN ENGINEERING FOR APPLIED MONOCULAR RECONSTRUCTION OF PARAMETRIC FACES
 
Domain Engineering for Applied Monocular Reconstruction of Parametric Faces
Domain Engineering for Applied Monocular Reconstruction of Parametric FacesDomain Engineering for Applied Monocular Reconstruction of Parametric Faces
Domain Engineering for Applied Monocular Reconstruction of Parametric Faces
 
2005 Web Content Mining 4
2005 Web Content Mining   42005 Web Content Mining   4
2005 Web Content Mining 4
 
Sadcw 7e chapter03-done(1)
Sadcw 7e chapter03-done(1)Sadcw 7e chapter03-done(1)
Sadcw 7e chapter03-done(1)
 
pam_1997
pam_1997pam_1997
pam_1997
 
IRJET - Airplane Crash Analysis and Prediction using Machine Learning
IRJET - Airplane Crash Analysis and Prediction using Machine LearningIRJET - Airplane Crash Analysis and Prediction using Machine Learning
IRJET - Airplane Crash Analysis and Prediction using Machine Learning
 
Decision Making in Product Design
Decision Making in Product DesignDecision Making in Product Design
Decision Making in Product Design
 
American Psychological Association (APA) information for .docx
American Psychological Association (APA) information for .docxAmerican Psychological Association (APA) information for .docx
American Psychological Association (APA) information for .docx
 
Your Parts Data and Why You NEED SPEEDcat!
Your Parts Data and Why You NEED SPEEDcat!Your Parts Data and Why You NEED SPEEDcat!
Your Parts Data and Why You NEED SPEEDcat!
 
Basics (Reports Writing 2)
Basics (Reports Writing 2)Basics (Reports Writing 2)
Basics (Reports Writing 2)
 
autonomous_vehicle_working_paper_01072020-_508_compliant.pdf
autonomous_vehicle_working_paper_01072020-_508_compliant.pdfautonomous_vehicle_working_paper_01072020-_508_compliant.pdf
autonomous_vehicle_working_paper_01072020-_508_compliant.pdf
 

Recently uploaded

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix
 
Azure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAzure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAndikSusilo4
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK
 
How to Remove Document Management Hurdles with X-Docs?
How to Remove Document Management Hurdles with X-Docs?How to Remove Document Management Hurdles with X-Docs?
How to Remove Document Management Hurdles with X-Docs?XfilesPro
 
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your BudgetHyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your BudgetEnjoy Anytime
 
Pigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent
 
Benefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other FrameworksBenefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other FrameworksSoftradix Technologies
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
Next-generation AAM aircraft unveiled by Supernal, S-A2
Next-generation AAM aircraft unveiled by Supernal, S-A2Next-generation AAM aircraft unveiled by Supernal, S-A2
Next-generation AAM aircraft unveiled by Supernal, S-A2Hyundai Motor Group
 
Key Features Of Token Development (1).pptx
Key  Features Of Token  Development (1).pptxKey  Features Of Token  Development (1).pptx
Key Features Of Token Development (1).pptxLBM Solutions
 

Recently uploaded (20)

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food Manufacturing
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
 
The transition to renewables in India.pdf
The transition to renewables in India.pdfThe transition to renewables in India.pdf
The transition to renewables in India.pdf
 
Vulnerability_Management_GRC_by Sohang Sengupta.pptx
Vulnerability_Management_GRC_by Sohang Sengupta.pptxVulnerability_Management_GRC_by Sohang Sengupta.pptx
Vulnerability_Management_GRC_by Sohang Sengupta.pptx
 
Azure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAzure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & Application
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
 
How to Remove Document Management Hurdles with X-Docs?
How to Remove Document Management Hurdles with X-Docs?How to Remove Document Management Hurdles with X-Docs?
How to Remove Document Management Hurdles with X-Docs?
 
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your BudgetHyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
 
Pigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping Elbows
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
 
Benefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other FrameworksBenefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other Frameworks
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Next-generation AAM aircraft unveiled by Supernal, S-A2
Next-generation AAM aircraft unveiled by Supernal, S-A2Next-generation AAM aircraft unveiled by Supernal, S-A2
Next-generation AAM aircraft unveiled by Supernal, S-A2
 
Key Features Of Token Development (1).pptx
Key  Features Of Token  Development (1).pptxKey  Features Of Token  Development (1).pptx
Key Features Of Token Development (1).pptx
 

Coreference Resolution

  • 1. Corefrence Resolution A Machine Learning Approach nicolas_ nicolov @ jdpa .com shumin. wu @ colorado .edu Shumin Wu Ph.D. Candidate in Computer Science University of Colorado at Boulder The Center for Spoken Language Research  1777 Exposition Drive Boulder, Colorado 80301, U.S.A. Nicolas Nicolov Senior Director, Science J.D. Power and Associates, McGraw-Hill Web Intelligence Division 4888 Pearl East Circle Boulder, CO 80301, U.S.A.
  • 2.
  • 3. Coreference Audi is an automaker that makes luxury cars and SUVs. The company was born in Germany . It was established by August Horch in 1910. Horch had previosly founded another company and his models were quite popular. Audi started with four cylinder models. By 1914, Horch 's new cars were racing and winning. August Horch left the Audi company in 1920 to take a position as an industry representative for the German motor vehicle industry federation. Currently Audi is a subsidiary of the Volkswagen group and produces cars of outstanding quality.
  • 4.
  • 5. MUC6 F-measure a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Count the number of corresponding links between mentions Precision = 4/5 Recall = 4/6 F-measure = 2* Precision * Recall/( Precision + Recall ) = 0.727
  • 6. MUC6 F-measure Degenerate Case a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Precision = N/A Recall = 0 F-measure = N/A a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Discounts single mention entities All mentions form individual singleton entities.
  • 7. MUC6 F-measure Degenerate Case 2 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Precision = 6/8 Recall = 1 F-measure = 0.857 !!! a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Does not adequately penalize dense links. All mentions form one big entity.
  • 8.
  • 9. B 3 F-measure a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: For each mention, compute the proportion of corresponding mentions between reference and system entity. Precision =1/9*( Recall = 1/9*( F-measure = 0.760 3/3 +3/3 +3/3 +1/3 +2/3 +2/3 +1/1 +2/2 +2/2 ) = 0.852 3/4 +3/4 +3/4 +1/4 +2/2 +2/2 +1/3 +2/3 +2/3 ) = 0.685
  • 10. B 3 F-measure Degenerate Case a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Precision = 1 Recall = 1/3 F-measure = 0.5 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 All mentions form individual singleton entities.
  • 11. B 3 F-measure Degenerate Case 2 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Precision = 1/9*(4/9+4/9+4/9+4/9+2/9+2/9+3/9+3/9+3/9) = 0.358 Recall = 1 F-measure = 0.527 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Which system entity maps to which reference entity? All mentions form one big entity.
  • 12.
  • 13. CEAF a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Find the one-to-one entity mapping between reference (R) and system (S) maximizing similarity measure
  • 14. CEAF Degenerate Case a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 All mentions form individual singleton entities.
  • 15. CEAF Degenerate Case 2 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 All mentions form one big entity.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27. Heuristic System: Results Order of clustering (local (mentions within sentence) to global, forward, and reverse direction) did not alter our results. Configuration MUC-F MUC-P MUC-R B 3 -F B 3 -P B 3 -R Unlinked entities -- -- 0 71.8 100 56.0 Single entity (w/ all mentions) 61.4 44.3 100 8.6 4.5 100 w/ editing distant 66.2 64.0 68.5 78.6 76.8 80.4 editing dist. at sentence level 64.8 63.7 66.0 78.7 77.8 79.4 w/o editing distant 70.0 75.8 64.9 83.0 86.8 79.5 Edit dist + cardinality match 70.4 78.7 63.7 83.7 89.2 78.8
  • 28.
  • 29.
  • 30. Robust Risk Minimization (RRM) RRM was proposed by Tong Zhang; best CoNLL’03 chunker. Separate positive and negative weights Multiplicative weight update
  • 31. Bell Tree Input sequence: a 1 , b 1 , b 2 , a 2 … Lots of states in the search space to explore! [ a 1 ] [a 1 , b 1 ] [a 1 ][ b 1 ] [a 1 ,b 1 , b 2 ] [a 1 ,b 1 ][ b 2 ] [a 1 , b 2 ][b 1 ] [a 1 ][b 1 , b 2 ] [a 1 ][b 1 ][ b 2 ] [a 1 ,b 1 ,b 2 , a 2 ] [a 1 ,b 1 ,b 2 ][ a 2 ] [a 1 , a 2 ] [b 1 ,b 2 ] [a 1 ][b 1 ,b 2 , a 2 ] [a 1 ][b 1 ,b 2 ][ a 2 ] [a 1 ,b 1 ][b 2 ][ a 2 ] [a 1 ,b 1 , a 2 ][b 2 ] [a 1 ,b 1 ][b 2 , a 2 ] [a 1 ,b 2 , a 2 ][b 1 ] [a 1 ,b 2 ][b 1 , a 2 ] [a 1 ,b 2 ][b 1 ][ a 2 ] [a 1 , a 2 ][b 1 ][b 2 ] [a 1 ][b 1 , a 2 ][b 2 ] [a 1 ][b 1 ][b 2 , a 2 ] [a 1 ][b 1 ][b 2 ] [ a 2 ]
  • 32.
  • 33. Bell Tree in Action Input sequence: a 1 , b 1 , b 2 , a 2 , c 1 … Coreference probability: [ a 1 ] p = 1 [a 1 , b 1 ] p=0.4 a 1 b 1 b 2 a 2 c 1 a 1 1 b 1 0.4 1 b 2 0.2 0.9 1 a 2 0.8 0.1 0.3 1 c 1 0.4 0.3 0.4 0.2 1 [a 1 ][ b 1 ] p=0.6 [a 1 ,b 1 , b 2 ] p=0.36 [a 1 ,b 1 ][ b 2 ] p=0.04 [a 1 , b 2 ][b 1 ] p=0.12 [a 1 ][b 1 , b 2 ] p=0.54 [a 1 ][b 1 ][ b 2 ] p=0.06 [a 1 ,b 1 ,b 2 , a 2 ] p=0.288 [a 1 ,b 1 ,b 2 ][ a 2 ] p=0.072 [a 1 , a 2 ] [b 1 ,b 2 ] p=0.432 [a 1 ][b 1 ,b 2 , a 2 ] p=0.162 [a 1 ][b 1 ,b 2 ][ a 2 ] p=0.108 [a 1 ,b 1 ,b 2 , a 2 , c 1 ] p=0.1152 [a 1 ,b 1 ,b 2 , a 2 ][ c 1 ] p=0.1728 [a 1 ,a 2, c 1 ][b 1 , b 2 ] p=0.1728 [a 1 ,a 2 ][b 1 , b 2, c 1 ] p=0.1728 [a 1 ,a 2 ][b 1 , b 2 ][ c 1 ] p=0.2592
  • 34.
  • 35.
  • 36.
  • 37.

Editor's Notes

  1. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  2. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  3. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  4. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  5. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  6. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  7. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  8. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  9. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  10. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  11. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  12. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  13. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  14. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  15. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  16. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  17. 100648 mentions 67038 coref groups longest mention: http://images.myride.com/images/non-vehicle/Misc/Shows/2007%20Sema/Jeep/Wrangler/OEM/JeepWranglerUltimatewith392HEMIfromMopar2_ (544x408).jpg CookingMethod 8 CameraFeature 9575 Vehicles.SUVs 2347 FoodFeature 348 Person 13505 Vehicles.Cars 10258 Units.Money 986 FoodPart 13 Vehicles.Trucks 467 Location 245 Units 1192 Time.Date 103 CameraPart 3240 Time.Month 171 Time.DaysOfTheWeek 49 Food 801 Marketing 124 Time.Year 1443 CookingTool 3 CarFeature 6119 Time.OClock 21 CarPart 15478 Vehicles 905 GeoPolitical 138 Units.Rate 764 GeoPolitical.City 360 Beverage 82 Time 924 Organization 8206 CameraAccessory 352 Meal 28 Units.Age 57 Facility 1052 Descriptor 13087 GeoPolitical.USStates 127 GeoPolitical.Countries 316 Time.Duration 540 Camera 5197 nutrient 247 GeoPolitical.Nationalities 195
  18. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  19. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  20. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  21. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  22. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  23. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  24. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  25. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  26. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  27. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  28. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  29. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  30. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  31. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
  32. 02/21/10 23:38 © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.