Like this presentation? Why not share!

# Dynamic Score Combination: A supervised and unsupervised score combination method

## on Jul 24, 2009

• 728 views

In two-class score-based problems the combination of scores from an ensemble of experts is generally used to obtain distributions for positive and negative patterns that exhibit a larger degree of ...

In two-class score-based problems the combination of scores from an ensemble of experts is generally used to obtain distributions for positive and negative patterns that exhibit a larger degree of separation than those of the scores to be combined. Typically, combination is carried out by a "static" linear combination of scores, where the weights are computed by maximising a
performance function. These weights are equal for all the patterns, as they are assigned to each of the expert to be combined. In this paper we propose a "dynamic" formulation where the weights are computed individually for each pattern. Reported results on a biometric dataset show the effectiveness of the proposed combination methodology with respect to "static" linear combinations and trained combination rules.

### Views

Total Views
728
Views on SlideShare
728
Embed Views
0

Likes
1
11
0

No embeds

### Report content

• Comment goes here.
Are you sure you want to

## Dynamic Score Combination: A supervised and unsupervised score combination methodPresentation Transcript

• Dynamic Score Combination a supervised and unsupervised score combination method R. Tronci, G. Giacinto, F. Roli DIEE - University of Cagliari, Italy Pattern Recognition and Applications Group http://prag.diee.unica.it MLDM 2009 - Leipzig, July 23-25, 2009
• Outline ! Goal of score combination mechanisms ! Dynamic Score Combination ! Experimental evaluation ! Conclusions Giorgio Giacinto MLDM 2009 - July 23-25, 2009 2
• Behavior of biometric experts Genuine scores should produce a positive outcome Impostor scores should produce a negative outcome th FNMRj (th) = \$ p(s j | s j ! positive)ds j = P(s j % th | s j ! positive) "# # FMRj (th) = \$ p(s j | s j ! negative)ds j = P(s j > th | s j ! negative) th Giorgio Giacinto MLDM 2009 - July 23-25, 2009 3
• Performance assessment ! True Positive Rate = 1 - FNMR Giorgio Giacinto MLDM 2009 - July 23-25, 2009 4
• Goal of score combination ! To improve system reliability, different experts are combined ! different sensors, different features, different matching algorithms ! Combination is typically performed at the matching score level Giorgio Giacinto MLDM 2009 - July 23-25, 2009 5
• Goal of score combination Combined score Giorgio Giacinto MLDM 2009 - July 23-25, 2009 6
• Goal of score combination ! The aim is to maximize the separation between classes e.g. (µ ) 2 gen ! µimp FD = " gen + " imp 2 2 ! Thus the distributions have to be shifted far apart, and the spread of the scores reduced Giorgio Giacinto MLDM 2009 - July 23-25, 2009 7
• Static combination ! Let E = {E1,E2,…Ej,…EN} be a set of N experts ! Let X = {xi} be the set of patterns ! Let fj ( ) be the function associated to expert Ej that produces a score sij = fj(xi) for each pattern xi Static linear combination N si* = # ! j " sij j =1 ! The weights are computed as to maximize some measure of class separability on a training set ! The combination is static with respect to the test pattern to be classified Giorgio Giacinto MLDM 2009 - July 23-25, 2009 8
• Dynamic combination The weights of the combination also depends on the test pattern to be classified N si* = # ! ij " sij j =1 The local estimation of combination parameters may yield better results than the global estimation, in terms of separation between the distributions of scores si* Giorgio Giacinto MLDM 2009 - July 23-25, 2009 9
• Estimation of the parameters for the dynamic combination ! Let us suppose without loss of generality s i1 ! s i2 ! ! ! siN ! The linear combination of three experts ! i1si1 + ! i 2 si 2 + ! i 3 si 3 ! ij "[ 0,1] can also be written as " i1si1 + si 2 + " i!3 si 3 ! which is equivalent to " i1si1 + " i!! si 3 !! 3 Giorgio Giacinto MLDM 2009 - July 23-25, 2009 10
• Estimation of the parameters for the dynamic combination ! This reasoning can be extended to N experts, so we can get ( ) si* = !i1 min sij + !i 2 max sij j j ( ) ! Thus, for each pattern we have to estimate two parameters ! If we set the constraint !i1 + !i 2 = 1 only one parameter has to be estimated and si* ! [minj(sij),maxj(sij)] Giorgio Giacinto MLDM 2009 - July 23-25, 2009 11
• Properties of the Dynamic Score Combination ( ) si* = !i max sij + (1 " !i ) min sij j j ( ) ! This formulation embeds the typical static combination rules #" J sij \$ min ( sij ) N j j =1 Linear combination !i = ( ) ( ) ! max sij \$ min sij j j 1 N ( ) " sij # min sij N j =1 j Mean rule !i = max ( s ) # min ( s ) ! ij ij j j ! Max rule for "i = 1 and Min rule for "i = 0 Giorgio Giacinto MLDM 2009 - July 23-25, 2009 12
• Properties of the Dynamic Score Combination ( ) si* = !i max sij + (1 " !i ) min sij j j ( ) ! This formulation also embeds the Dynamic Score Selection (DSS) "1 if xi belongs to the positive class !i = # \$0 if xi belongs to the negative class ! DSS clearly maximize class separability if the estimation of the class of xi is reliable ! e.g., a classifier trained on the outputs of the experts E Giorgio Giacinto MLDM 2009 - July 23-25, 2009 13
• Supervised estimation of "i ( ) si* = !i max sij + (1 " !i ) min sij j j ( ) ! "i = P(pos|xi,E) P(pos|xi,E) can be estimated by a classifier trained on the outputs of the experts E ! "i is estimated by a supervised procedure ! This formulation can also be seen as a soft version of DSS ! P(pos|xi,E) accounts for the uncertainty in class estimation Giorgio Giacinto MLDM 2009 - July 23-25, 2009 14
• Unsupervised estimation of "i ( ) si* = !i max sij + (1 " !i ) min sij j j ( ) ! "i is estimated by an unsupervised procedure ! the estimation does not depend on a training set 1 N Mean rule !i = " sij N j =1 Max rule !i = max sij j ( ) Min rule !i = min sij j ( ) Giorgio Giacinto MLDM 2009 - July 23-25, 2009 15
• Dataset ! The dataset used is the Biometric Scores Set Release 1 of the NIST http://www.itl.nist.gov/iad/894.03/biometricscores/ ! This dataset contains scores from 4 experts related to face and fingerprint recognition systems. ! The experiments were performed using all the possible combinations of 3 and 4 experts. ! The dataset has been divided into four parts, each one used for training and the remaining three for testing Giorgio Giacinto MLDM 2009 - July 23-25, 2009 16
• Experimental Setup ! Experiments aimed at assessing the performance of ! The unsupervised Dynamic Score Combination (DSC) ! "i estimated by the Mean, Max, and Min rules ! The supervised Dynamic Score Combination ! "i estimated by k-NN, LDC, QDC, and SVM classifiers ! Comparisons with ! The Ideal Score Selector (ISS) ! The Optimal static Linear Combination (Opt LC) ! The Mean, Max, and Min rules ! The linear combination where coefficients are estimated by the LDA Giorgio Giacinto MLDM 2009 - July 23-25, 2009 17
• Performance assessment ! Area Under the ROC Curve (AUC) ! Equal Error Rate (ERR) µ gen " µimp ! d! = # gen # imp 2 2 + 2 2 ! FNMR at 1% and 0% FMR ! FMR at 1% and 0% FNMR Giorgio Giacinto MLDM 2009 - July 23-25, 2009 18
• Combination of three experts AUC EER d’ ISS 1.0000 (±0.0000) 0.0000 (±0.0000) 25.4451 (±8.7120) Opt LC 0.9997 (±0.0004) 0.0050 (±0.0031) 3.1231 (±0.2321) Mean 0.9982 (±0.0013) 0.0096 (±0.0059) 3.6272 (±0.4850) Max 0.9892 (±0.0022) 0.0450 (±0.0048) 3.0608 (±0.3803) Min 0.9708 (±0.0085) 0.0694 (±0.0148) 2.0068 (±0.1636) DSC Mean 0.9986 (±0.0011) 0.0064 (±0.0030) 3.8300 (±0.5049) DSC Max 0.9960 (±0.0015) 0.0214 (±0.0065) 3.8799 (±0.2613) DSC Min 0.9769 (±0.0085) 0.0634 (±0.0158) 2.3664 (±0.2371) LDA 0.9945 (±0.0040) 0.0296 (±0.0123) 2.3802 (±0.2036) DSC k-NN 0.9987 (±0.0016) 0.0104 (±0.0053) 6.9911 (±0.9653) DSC ldc 0.9741 (±0.0087) 0.0642 (±0.0149) 2.7654 (±0.2782) DSC qdc 0.9964 (±0.0039) 0.0147 (±0.0092) 9.1452 (±3.1002) DSC svm 0.9996 (±0.0004) 0.0048 (±0.0026) 4.8972 (±0.4911) Giorgio Giacinto MLDM 2009 - July 23-25, 2009 19
• DSC Mean Vs. Mean rule Combination of three experts DSC Mean AUC !!0.9991 EER !!0.0052 d' !!!4.4199 Mean rule AUC !!0.9986 EER !!0.0129 d' !!!4.0732 Giorgio Giacinto MLDM 2009 - July 23-25, 2009 20
• Unsupervised DSC Vs. fixed rules AUC Giorgio Giacinto MLDM 2009 - July 23-25, 2009 21
• Unsupervised DSC Vs. fixed rules EER Giorgio Giacinto MLDM 2009 - July 23-25, 2009 22
• Unsupervised DSC Vs. fixed rules FMR at 0% FNMR Giorgio Giacinto MLDM 2009 - July 23-25, 2009 23
• DSC Mean Vs. supervised DSC AUC Giorgio Giacinto MLDM 2009 - July 23-25, 2009 24
• DSC Mean Vs. supervised DSC EER Giorgio Giacinto MLDM 2009 - July 23-25, 2009 25
• DSC Mean Vs. supervised DSC FMR at 0% FNMR Giorgio Giacinto MLDM 2009 - July 23-25, 2009 26
• Conclusions ! The Dynamic Score Combination mechanism embeds different combination modalities ! Experiments show that the unsupervised DSC usually outperforms the related “fixed” combination rules ! The use of a classifier in the supervised DSC allows attaining better performance, at the expense of increased computational complexity ! Depending on the classifier, performance are very close to those of the optimal linear combiner Giorgio Giacinto MLDM 2009 - July 23-25, 2009 27