2. Contents
I. Is forensic science a science?
II. Cognitive Bias
III. Efforts for Quantitative Representation in Forensic Science
IV. Efforts in NFS for Audio Forensics
5. Scientific Thinking
Science Pseudoscience Religion
Willingness to change with new
evidence
Fixed ideas
Fixed incongruous & antiquated
ideas
Ruthless peer review No peer review Unfalsifiable
Takes account of all new
discoveries
Selects only favorable
discoveries
Selects favorable discoveries
blames rest on demons
Invites criticism Sees criticism as conspiracy Blasphemous!
Verifiable results Non-repeatable results Results same as random chance
Limits claims of usefulness
Claims of widespread
usefulness
Claims complete & Exclusive
usefulness
Accurate measurement “Ball-park” measurement Blind faith
6. There are no absolute and fixed truths.
• The scientific method as an ongoing
process must be guided by the best
available evidence.
• Errors in scientific measurements
always exist.
• The history of science has developed
by finding paradox against paradigm,
and there is absolute and fixed
truths.
Quantitative, accurate, no error
Public perception of “scientific”
BUT the results of “scientific way”
Quantitation based on reliability
Accuracy is a measure with errors
Difference
- F. W. Nietzsche
7. • The validity of the application of the
technology
• Ensuring the reliability that can be
measured consistently.
8. NAS Report, 2009
• Forensic science evidence other than DNA analysis has
not secured scientific validity for the purpose of
individual identification.
• Issues recommendations for the challenges facing the
forensic science community and for the advancement
of forensic science and strengthening of proof.
National Research Council, Strengthening Forensic Science in the United States:
A Path Forward (National Academies Press, 2009)
11. • The National Registry of Exonerations
• As of 2017.1., about 460 cases
• Innocence Preoject
• As of 2011., about 300 cases in DNA analysis
False or Misleading Forensic Evidence
Comparison of two patterns or samples
Questioned Documents
Trace Evidence (hair, fiber, paint etc.)
Biometric Information (fingerprint, footprint, bite mark etc.)
Blood Stain Analysis
Arson
DNA
CSI
12. [1] Saul M. Kassin, Itiel E. Dror, Jeff Kukucka, The forensic confirmation bias: Problems, perspectives, and
proposed solutions, Journal of Applied Research in Memory and Cognition, 2(1), pp.42-52 (2013).
• Michele Triplett, Errors in forensics: Cause(s) and solutions, Journal of Applied Research in Memory and Cognition, 2(1),
pp.63-64 (2013).
• Ralph Norman Haber and Lyn Haber, The culture of science: Bias and forensic evidence, Journal of Applied Research in
Memory and Cognition, 2(1), pp.65-67 (2013).
[2] Glinda S. Cooper, Vanessa Meterko, Cognitive Bias research in forensic science: A systematic review, Forensic
Science International, 297, pp.35-46 (2019).
The Cognitive Bias of Forensic Examiner
Effect of limited analysis time and
prior information on results
13. • Classic confirmation biases: a psychological perspective
1. Perceptual and cognitive effects
2. Social perception effects
3. Cognitive and motivational sources of bias
• The forensic confirmation bias
• The class of effects through which an individual’s preexisting beliefs, expectations, motives, and
situational context influence the collection, perception, and interpretation of evidence during the
course of a criminal case
1. Context effects on forensic judgments
2. Elasticity of forensic evidence
3. Bias and self-insight
4. Null effects from the Netherlands
5. Many efforts to avoid physical contamination but less attention to “psychological contamination”
Using computerized system: examiners are more likely to make false positive errors on candidates
on the top of the list and false negative errors on those near the bottom
The Cognitive Bias of Forensic Examiner
14. 1. Ethics violation
A. Fabricated prints
B. Dry benching (estimating results without completing an examination)
C. Intentional erroneous results
D. Covering up mistakes
2. Honest errors
A. Lack of training and mentoring
B. Feeling pressure to complete work or being overwhelmed with work
C. Administrative errors or complacency in one’s work
3. Biased oversight
Types of Errors in Forensic Science
Jon S. Byrd, Confirmation Bias, Ethics, and Mistakes in Forensics, Journal of Forensic Identification, 56(4), pp.511-525 (2006).
15. 1. Across many domains, experts are often overconfident in their abilities
2. The courts, for the most part, have blindly accepted forensic science evidence
without much scrutiny
3. Errors are often not apparent in the forensic sciences because ground truth is
often not known as a matter of certainty
4. Many forensic examiners work for police and appear in court as advocates for
the prosecution
5. Many forensic examiners consider themselves objective and immune to bias
Background of the Appearance of a “Wrongful”
Forensic Science
Saul M. Kassin, Itiel E. Dror, Jeff Kukucka, The forensic confirmation bias: Problems, perspectives, and proposed solutions,
Journal of Applied Research in Memory and Cognition, 2(1), pp.42-52 (2013).
16. Did forensic scientists recognize that they can
cognitively bias themselves?
Jeff Kukucka, Saul M. Kassina, Patricia A. Zapf, Itiel E. Dror, Cognitive Bias and Blindness:A Global Survey of
Forensic Science Examiners, Journal of Applied Research in Memory and Cognition, 6(4), pp.452-459(2017).
Participants: 403 professional forensic examiners
from 21 different countries
Age: 44.02 ± 11.39 years old
Experience: 14.4 ± 9.6 years
Working amount:
Average 1,000 cases in their career
(IQR=487.75 ~ 4,875)
*For 370 examiners
Testimony in court:
Average 25 times
(IQR=7 ~ 80.75)
*For 396 examiners
Domain %
Biology, DNA 24.07
Fingerprint 14.64
QD 8.68
Toxicology 6.20
Firearm/tool
mark
5.96
Multiple 17.62
Degree %
College 42.43
Masters 38.71
Ph.D. 10.67
Gender #
Men 183
Women 219
N/A 1
Work for %
Exclusively for the
prosecution
28.29
Mostly for the
prosecution
46.40
Exclusively for the
defense
0.25
Mostly for the
defense
0.74
Country %
US 82.38
Outside US 17.62
Laboratory
size
%
21+
employees
57.57
Under 5 8.44
Alone 6.95
17. "In recent years, there has been some debate over whether forensic examiners are subconsciously influenced by
prior beliefs and expectations formed on the basis of contextual information (e.g., a detective’s opinion, evidence
from other forensic domains, a suspect’s criminal history, a confession, an eyewitness) that is irrelevant to the
forensic samples they are evaluating. This phenomenon has been referred to as cognitive bias.”
Did forensic scientists recognize that they
can cognitively bias themselves?
18.
19.
20.
21. • Examiners should work “linear” rather than “circular”
Initially examining the evidence from the crime scene
Documenting their findings before making comparisons against a target
• Blind test under shielding various prior information
Forensic examiners must be isolated from undue influences such as direct contact with the investigating
officer, the victims and their families, and other irrelevant information
• The verification of forensic decisions
A more controlled process in which blind and double-blind procedures are used whenever possible
• Results from AI(Artificial Intelligence) or computerized system : the order of entries should be
randomized
• Forensic science education and certification to include training in basic psychology that is relevant
to forensic work
Perception, judgment and decision making, and social influence
How to reduce bias: proposed reforms
23. Efforts to Improve the Reliability
• After the 2009 NAS Report,
systematic research conducted
by forensic science institution
• Many efforts for quantification
of forensic result : developing
criteria
• Many studies on cognitive
psychology problems related
to probabilistic expression of
quantification
24. Efforts to Evaluate the Forensic Result
Likelihood Ratio (LR)
Verbal Expression
ENFSI SWGDAM
1 Do not support Uninformative
2 ~ 10 Weak support
Limited support
10 ~ 100 Moderate support
100 ~ 1,000 Moderately strong support
Moderate support
1,000 ~ 10,000 Strong support
10,000 ~ 1,000,000 Very strong support Strong support
1,000,000 and above Extremely strong support Very strong support
30. Real Cases
• □: “이해를 했어요?” ( “Understand?” in Korean)
#1
#2
The 3rd Formant
The 2nd Formant
The 1st Formant
31. The Limitation of Voiceprint Analysis
Ambiguity in voiceprint analysis
• M. A. Young et al., Effects of context on talker identification, JASA 42, 1250-
1254(1967).
• K. N. Stevens et al., Speaker authentication and identification: A comparison of
spectrographic and auditory presentations of speech material, JASA 44, 1596-
1607(1987).
• O. Tosi et al., Experiment on voice identification, JASA 51, 2030-2043 (1972).
• B. E. Koenig, Spectrographic voice identification: A forensic survey, JASA 79, 288-
2090 (1986).
Test results according to the voiced information and data acquisition conditions
- Specified paragraph
- Listen with voiceprint
Errors among the forensic examiners
32. Voice Comparison Based on
Voiceprint Analysis
• Method
Structural characteristics
Habit of vocalization
Linguistic characteristics
Inter speaker variation is much larger than intra speaker variation
• Limitation
• The length of speech data at least 7s
• Language, vocalization condition, text dependency
• Results variance by examiner’s proficiency
• Limitation to numerical representation of error rate and accuracy
• Limitation to improve promptness, vulnerable to large data analysis
33. How Could We Deal with Following Problems?
Images
[1] 「名探偵コナン」の原作公式サイト, https://www.conan-portal.com/#newsContent
[2] 조선일보 연예, http://news.chosun.com/site/data/html_dir/2018/09/06/2018090603647.html
Video Clip
[3] Youtube, https://www.youtube.com/watch?v=CgX4uJSj00Y, Mission Impossible 3, 2006.
34.
35. The Blind Test Results in NFS for Audio
Forensics
• Result of Forensic Examiners: 3 persons
• False Rejection Rate, FRR
• False Acceptance Rate, FAR
Total (200
questions)
Acceptance of decision difficulty
Non-acceptance of decision
difficulty
Correct-answer
rate
Wrong-answer
rate
Decision
difficulty
Correct-answer
rate
Wrong-answer
rate
1-person test 56.00 6.33 37.67 90.44 9.56
2-person test 43,67 1.50 54.83 96.65 3.35
3-person
unanimous
agreement
39.00 0.50 60.50 98.73 1.27
36. Result Analysis
• Characteristics of the forensic examiner’s blind test
FRR > FAR
FAR → 0
The post-test exchange of opinions
influenced their decision
1-person test
3-person unanimous
agreement
2-person test
Total
37. The Blind Test Results of Laypersons
• Auditory test by laypersons
Total
Total
Age
Gender
20s
30s
40s
50s
M
F
Correct Incorrect N/A
Acceptance of decision difficulty No acceptance of
decision difficulty
Correct Incorrect
38. Result Anaysis
For forensic examiner(FE)’s unanimous
agreement cases
FRR > FAR for laypersons
• Characteristics of the laypersons’ blind test
Total
M
F
Total
FE’sunanimousQ
M
F
Total
39. Machine Learning Based Statistical
Voice Comparison
• Low dependency on language, vocalization condition, text
• Less variance among examiners
• Possible to numerical representation of error rate and accuracy
• Possible to large data analysis
41. NFS Own Developed System
For 80 samples nxn comparison : 1.5hrs (in 2019)
42. Decision Level
Under 10% error rate
Possible to suggest for the
decision difficulty in
spectrogram-based analysis
Possible to improve by
following developed algorithm
89.1 92.2
43. Voice-Phishing Fraud Group
• In 2017, a voice-phishing fraud
group was arrested
With large numbers of voice data
By using machine-learning based
system
Revealing another guilt
발견법, 불충분한 시간이나 정보로 인하여 합리적인 판단을 할 수 없거나, 체계적이면서 합리적인 판단이 굳이 필요하지 않은 상황에서 사람들이 빠르게 사용할 수 있는 어림짐작의 방법
인간의 의사결정 과정
태생적 불완전한 정보처리 능력
오류 최소화
시행착오
찍기의 법칙
False or misleading forensic evidence was a contributing factor in 300 cases
examiners expressed a personal interest in catching criminals and solving crimes, which some reported as more pronounced in serious and high-profile cases
that suggest a list of candidates for the human examiner to consider
all illustrated through the use of forensic case materials.