BCC 2003 - Reintroducing Watchlist Statistics

580 views

Published on

This presentation "reintroduces" watchlist statistics to the biometrics community at the 2004 Biometric Consortium Conference. At this time, many biometric studies and recommendations incorrectly referenced closed set identification statistics for open set applications.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
580
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

BCC 2003 - Reintroducing Watchlist Statistics

  1. 1. Reintroducing Watchlist Statistics Duane Blackburn Federal Bureau of Investigation dblackburn@fbiacademy.edu Biometrics Technology
  2. 2. Face Recognition Debate Fall 2001 Could the hijackers have been spotted? Biometrics Technology
  3. 3. Face Recognition Debate Fall 2001 Can it track me? Gregory Tara Kaye Nancy DOB: 4/21/74 Dorsey Leflore Dorothy Divorced Mother of 2 Shedd 2002 Income: $45,328 Matthew Millner Dennis Ricks Joe Grammer Gerald Elinor Galbraith Pardini Allan Olliff Wilfred Moesch Clayton Tayler Weddell Persky Biometrics Technology Nicolas Marocco
  4. 4. Face Recognition Debate Fall 2001 Or will this happen? Osama bin Laden Biometrics Technology
  5. 5. Why the confusion? • Confusing – Verification vs. Identification • incorrect usage of statistics – Identification (open set) vs. Identification (closed set) – False alarm rate, false accept rate, false match rate, etc. Biometrics Technology
  6. 6. Statistics Shortcomings • ROC (Verification) – Making a claim to your identity • CMC (Identification) – Each probe is in the database • Are measuring closed set identification • Neither relates to the problem! – Solution: develop statistics that measure this problem. Biometrics Technology
  7. 7. Chronologically • Face Recognition at a Chokepoint - Scenario Evaluation Results. http://www.dodcounterdrug.com/facialrecognition • FRVT 2002. http://www.frvt.org. • Both available in the Biometrics Catalog, http://www.biometricscatalog.org. Biometrics Technology
  8. 8. First Step • Forget what you know! Biometrics Technology
  9. 9. Baseline Performance Face Recognition Vendor Test 2002 (FRVT 2002) - http://www.frvt.org Biometrics Technology
  10. 10. Three Tasks • Verification – Are you who you say you are? • Watchlist – Are you in my database? If so, who are you? • Identification – You are in my database, can I find you? Biometrics Technology
  11. 11. Verification Threshold = 0.8 Claim Response = 0.75 Threshold = 0.7 Claim Response = 0.65 Threshold = 0.6 • Determines if the claimed identity of a face is correct. • Metrics: – Probability of Correct Verification • 1- (False Reject Rate) – False Accept Biometrics Technology
  12. 12. Verification Case #1 - Want a low false accept rate False Accept Rate = 0.1% Verification 1 0.9 Probability of Correct Verification 0.8 0.7 Cognitec 0.6 0.5 Eyematic 0.4 Identix 0.3 0.2 0.1 0 1 False Accept Rate = 0.1% Biometrics Technology
  13. 13. Verification Case #2 - Want a high verification rate Probability of Verification = 95% Verification Probability of Correct Verification = Cognitec 1 Eyematic Identix 95% 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Accept Rate Biometrics Technology
  14. 14. Verification ROC Verification Probability of Correct Verification = 95% Cognitec 1 Eyematic Identix 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Accept Rate Verification 1 Probability of Correct 0.8 Verification 0.6 Cognitec Eyematic 0.4 Identix 0.2 0 1 False Accept Rate = 0.1% Biometrics Technology
  15. 15. Watchlist Threshold = 0.8 0.75 Threshold = 0.7 0.65 Threshold = 0.62 Database 0.60 0.55 0.25 Biometrics Technology
  16. 16. Watchlist Threshold = 0.8 0.75 Threshold = 0.7 Database 0.65 0.70 0.25 0.55 Biometrics Technology
  17. 17. Watchlist Case #1 - Want a high detect and identification rate Detect and Identify = 75% Watchlist Detect and Identification Rate = Cognitec 1 Eyematic 75% Identix 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Alarm Rate Biometrics Technology
  18. 18. Watchlist Case #2 - Want a low false alarm rate False Alarm Rate = 0.1% Watchlist Detection and Identification Rate 1 0.9 0.8 0.7 Cognitec 0.6 0.5 Eyematic 0.4 Identix 0.3 0.2 0.1 0 1 False Alarm Rate = 0.1% Biometrics Technology
  19. 19. Watchlist Watchlist Detect and Identification Rate = 75% Cognitec 1 Eyematic Identix 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Alarm Rate Watchlist 1 Detection and Identification 0.8 0.6 Cognitec Rate Eyematic 0.4 Identix 0.2 0 1 False Alarm Rate = 0.1% Biometrics Technology
  20. 20. Identification 0.75 Database 0.65 0.80 0.70 0.25 0.55 Biometrics Technology
  21. 21. Identification Database That’s watchlist, not identification! Biometrics Technology
  22. 22. Identification • I know you are in the database, but who are you? – Closed universe test. • Compares your picture to all pictures in the database and lists them according to similarity. • Metrics: – How often is the top answer correct? – How often is the answer in the top 2? – How often is the answer in the top 3? – How often is the answer in the top n? Biometrics Technology
  23. 23. Identification Biometrics Technology
  24. 24. Three Tasks • Verification – Are you who you say you are? • Watchlist – Are you in my database? If so, who are you? • Identification – You are in my database, can I find you? Make sure you use the correct terms and the correct statistics! Biometrics Technology
  25. 25. Government Biometric References • Biometrics Catalog - http://www.biometricscatalog.org • FRVT - http://www.frvt.org • Biometric Consortium – http://www.biometrics.org Biometrics Technology

×