1. Corefrence Resolution A Machine Learning Approach nicolas_ nicolov @ jdpa .com shumin. wu @ colorado .edu Shumin Wu Ph.D. Candidate in Computer Science University of Colorado at Boulder The Center for Spoken Language Research 1777 Exposition Drive Boulder, Colorado 80301, U.S.A. Nicolas Nicolov Senior Director, Science J.D. Power and Associates, McGraw-Hill Web Intelligence Division 4888 Pearl East Circle Boulder, CO 80301, U.S.A.
2.
3. Coreference Audi is an automaker that makes luxury cars and SUVs. The company was born in Germany . It was established by August Horch in 1910. Horch had previosly founded another company and his models were quite popular. Audi started with four cylinder models. By 1914, Horch 's new cars were racing and winning. August Horch left the Audi company in 1920 to take a position as an industry representative for the German motor vehicle industry federation. Currently Audi is a subsidiary of the Volkswagen group and produces cars of outstanding quality.
4.
5. MUC6 F-measure a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Count the number of corresponding links between mentions Precision = 4/5 Recall = 4/6 F-measure = 2* Precision * Recall/( Precision + Recall ) = 0.727
6. MUC6 F-measure Degenerate Case a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Precision = N/A Recall = 0 F-measure = N/A a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Discounts single mention entities All mentions form individual singleton entities.
7. MUC6 F-measure Degenerate Case 2 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Precision = 6/8 Recall = 1 F-measure = 0.857 !!! a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Does not adequately penalize dense links. All mentions form one big entity.
8.
9. B 3 F-measure a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: For each mention, compute the proportion of corresponding mentions between reference and system entity. Precision =1/9*( Recall = 1/9*( F-measure = 0.760 3/3 +3/3 +3/3 +1/3 +2/3 +2/3 +1/1 +2/2 +2/2 ) = 0.852 3/4 +3/4 +3/4 +1/4 +2/2 +2/2 +1/3 +2/3 +2/3 ) = 0.685
10. B 3 F-measure Degenerate Case a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Precision = 1 Recall = 1/3 F-measure = 0.5 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 All mentions form individual singleton entities.
11. B 3 F-measure Degenerate Case 2 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Precision = 1/9*(4/9+4/9+4/9+4/9+2/9+2/9+3/9+3/9+3/9) = 0.358 Recall = 1 F-measure = 0.527 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Which system entity maps to which reference entity? All mentions form one big entity.
12.
13. CEAF a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: Find the one-to-one entity mapping between reference (R) and system (S) maximizing similarity measure
14. CEAF Degenerate Case a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 All mentions form individual singleton entities.
15. CEAF Degenerate Case 2 a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 Reference: System output: a 1 a 2 a 3 a 4 b 1 b 2 c 1 c 2 c 3 All mentions form one big entity.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27. Heuristic System: Results Order of clustering (local (mentions within sentence) to global, forward, and reverse direction) did not alter our results. Configuration MUC-F MUC-P MUC-R B 3 -F B 3 -P B 3 -R Unlinked entities -- -- 0 71.8 100 56.0 Single entity (w/ all mentions) 61.4 44.3 100 8.6 4.5 100 w/ editing distant 66.2 64.0 68.5 78.6 76.8 80.4 editing dist. at sentence level 64.8 63.7 66.0 78.7 77.8 79.4 w/o editing distant 70.0 75.8 64.9 83.0 86.8 79.5 Edit dist + cardinality match 70.4 78.7 63.7 83.7 89.2 78.8
28.
29.
30. Robust Risk Minimization (RRM) RRM was proposed by Tong Zhang; best CoNLL’03 chunker. Separate positive and negative weights Multiplicative weight update
31. Bell Tree Input sequence: a 1 , b 1 , b 2 , a 2 … Lots of states in the search space to explore! [ a 1 ] [a 1 , b 1 ] [a 1 ][ b 1 ] [a 1 ,b 1 , b 2 ] [a 1 ,b 1 ][ b 2 ] [a 1 , b 2 ][b 1 ] [a 1 ][b 1 , b 2 ] [a 1 ][b 1 ][ b 2 ] [a 1 ,b 1 ,b 2 , a 2 ] [a 1 ,b 1 ,b 2 ][ a 2 ] [a 1 , a 2 ] [b 1 ,b 2 ] [a 1 ][b 1 ,b 2 , a 2 ] [a 1 ][b 1 ,b 2 ][ a 2 ] [a 1 ,b 1 ][b 2 ][ a 2 ] [a 1 ,b 1 , a 2 ][b 2 ] [a 1 ,b 1 ][b 2 , a 2 ] [a 1 ,b 2 , a 2 ][b 1 ] [a 1 ,b 2 ][b 1 , a 2 ] [a 1 ,b 2 ][b 1 ][ a 2 ] [a 1 , a 2 ][b 1 ][b 2 ] [a 1 ][b 1 , a 2 ][b 2 ] [a 1 ][b 1 ][b 2 , a 2 ] [a 1 ][b 1 ][b 2 ] [ a 2 ]
32.
33. Bell Tree in Action Input sequence: a 1 , b 1 , b 2 , a 2 , c 1 … Coreference probability: [ a 1 ] p = 1 [a 1 , b 1 ] p=0.4 a 1 b 1 b 2 a 2 c 1 a 1 1 b 1 0.4 1 b 2 0.2 0.9 1 a 2 0.8 0.1 0.3 1 c 1 0.4 0.3 0.4 0.2 1 [a 1 ][ b 1 ] p=0.6 [a 1 ,b 1 , b 2 ] p=0.36 [a 1 ,b 1 ][ b 2 ] p=0.04 [a 1 , b 2 ][b 1 ] p=0.12 [a 1 ][b 1 , b 2 ] p=0.54 [a 1 ][b 1 ][ b 2 ] p=0.06 [a 1 ,b 1 ,b 2 , a 2 ] p=0.288 [a 1 ,b 1 ,b 2 ][ a 2 ] p=0.072 [a 1 , a 2 ] [b 1 ,b 2 ] p=0.432 [a 1 ][b 1 ,b 2 , a 2 ] p=0.162 [a 1 ][b 1 ,b 2 ][ a 2 ] p=0.108 [a 1 ,b 1 ,b 2 , a 2 , c 1 ] p=0.1152 [a 1 ,b 1 ,b 2 , a 2 ][ c 1 ] p=0.1728 [a 1 ,a 2, c 1 ][b 1 , b 2 ] p=0.1728 [a 1 ,a 2 ][b 1 , b 2, c 1 ] p=0.1728 [a 1 ,a 2 ][b 1 , b 2 ][ c 1 ] p=0.2592