This document discusses various topics related to forensic science, including whether forensic science is truly a science, cognitive biases that can affect forensic examiners, efforts to introduce quantitative methods and reduce bias, and developments in audio forensics. Specifically, it addresses the scientific method, cognitive biases, likelihood ratios as a quantitative measure, blind testing techniques, and the analysis of voiceprints in audio forensics cases.

1. Forensic Science
2022-10-26
For Indonesia
Oc-Yeub Jeon
What is the Scientific way?

2. Contents
I. Is forensic science a science?
II. Cognitive Bias
III. Efforts for Quantitative Representation in Forensic Science
IV. Efforts in NFS for Audio Forensics

3. Is Forensic Science
a Science?

4. Scientific Method as an Ongoing Process

5. Scientific Thinking
Science Pseudoscience Religion
Willingness to change with new
evidence
Fixed ideas
Fixed incongruous & antiquated
ideas
Ruthless peer review No peer review Unfalsifiable
Takes account of all new
discoveries
Selects only favorable
discoveries
Selects favorable discoveries
blames rest on demons
Invites criticism Sees criticism as conspiracy Blasphemous!
Verifiable results Non-repeatable results Results same as random chance
Limits claims of usefulness
Claims of widespread
usefulness
Claims complete & Exclusive
usefulness
Accurate measurement “Ball-park” measurement Blind faith

6. There are no absolute and fixed truths.
• The scientific method as an ongoing
process must be guided by the best
available evidence.
• Errors in scientific measurements
always exist.
• The history of science has developed
by finding paradox against paradigm,
and there is absolute and fixed
truths.
Quantitative, accurate, no error
Public perception of “scientific”
BUT the results of “scientific way”
Quantitation based on reliability
Accuracy is a measure with errors
Difference
- F. W. Nietzsche

7. • The validity of the application of the
technology
• Ensuring the reliability that can be
measured consistently.

8. NAS Report, 2009
• Forensic science evidence other than DNA analysis has
not secured scientific validity for the purpose of
individual identification.
• Issues recommendations for the challenges facing the
forensic science community and for the advancement
of forensic science and strengthening of proof.
National Research Council, Strengthening Forensic Science in the United States:
A Path Forward (National Academies Press, 2009)

9. Cognitive Bias in
Forensic Science

10. Heuristic Theory
Decision Making
Fast and Efficient way.
BUT there are many cognitive biased results.
Many
Information
Limited Time

11. • The National Registry of Exonerations
• As of 2017.1., about 460 cases
• Innocence Preoject
• As of 2011., about 300 cases in DNA analysis
False or Misleading Forensic Evidence
Comparison of two patterns or samples
Questioned Documents
Trace Evidence (hair, fiber, paint etc.)
Biometric Information (fingerprint, footprint, bite mark etc.)
Blood Stain Analysis
Arson
DNA
CSI

12. [1] Saul M. Kassin, Itiel E. Dror, Jeff Kukucka, The forensic confirmation bias: Problems, perspectives, and
proposed solutions, Journal of Applied Research in Memory and Cognition, 2(1), pp.42-52 (2013).
• Michele Triplett, Errors in forensics: Cause(s) and solutions, Journal of Applied Research in Memory and Cognition, 2(1),
pp.63-64 (2013).
• Ralph Norman Haber and Lyn Haber, The culture of science: Bias and forensic evidence, Journal of Applied Research in
Memory and Cognition, 2(1), pp.65-67 (2013).
[2] Glinda S. Cooper, Vanessa Meterko, Cognitive Bias research in forensic science: A systematic review, Forensic
Science International, 297, pp.35-46 (2019).
The Cognitive Bias of Forensic Examiner
Effect of limited analysis time and
prior information on results

13. • Classic confirmation biases: a psychological perspective
1. Perceptual and cognitive effects
2. Social perception effects
3. Cognitive and motivational sources of bias
• The forensic confirmation bias
• The class of effects through which an individual’s preexisting beliefs, expectations, motives, and
situational context influence the collection, perception, and interpretation of evidence during the
course of a criminal case
1. Context effects on forensic judgments
2. Elasticity of forensic evidence
3. Bias and self-insight
4. Null effects from the Netherlands
5. Many efforts to avoid physical contamination but less attention to “psychological contamination”
Using computerized system: examiners are more likely to make false positive errors on candidates
on the top of the list and false negative errors on those near the bottom
The Cognitive Bias of Forensic Examiner

14. 1. Ethics violation
A. Fabricated prints
B. Dry benching (estimating results without completing an examination)
C. Intentional erroneous results
D. Covering up mistakes
2. Honest errors
A. Lack of training and mentoring
B. Feeling pressure to complete work or being overwhelmed with work
C. Administrative errors or complacency in one’s work
3. Biased oversight
Types of Errors in Forensic Science
Jon S. Byrd, Confirmation Bias, Ethics, and Mistakes in Forensics, Journal of Forensic Identification, 56(4), pp.511-525 (2006).

15. 1. Across many domains, experts are often overconfident in their abilities
2. The courts, for the most part, have blindly accepted forensic science evidence
without much scrutiny
3. Errors are often not apparent in the forensic sciences because ground truth is
often not known as a matter of certainty
4. Many forensic examiners work for police and appear in court as advocates for
the prosecution
5. Many forensic examiners consider themselves objective and immune to bias
Background of the Appearance of a “Wrongful”
Forensic Science
Saul M. Kassin, Itiel E. Dror, Jeff Kukucka, The forensic confirmation bias: Problems, perspectives, and proposed solutions,
Journal of Applied Research in Memory and Cognition, 2(1), pp.42-52 (2013).

16. Did forensic scientists recognize that they can
cognitively bias themselves?
Jeff Kukucka, Saul M. Kassina, Patricia A. Zapf, Itiel E. Dror, Cognitive Bias and Blindness:A Global Survey of
Forensic Science Examiners, Journal of Applied Research in Memory and Cognition, 6(4), pp.452-459(2017).
Participants: 403 professional forensic examiners
from 21 different countries
 Age: 44.02 ± 11.39 years old
 Experience: 14.4 ± 9.6 years
 Working amount:
Average 1,000 cases in their career
(IQR=487.75 ~ 4,875)
*For 370 examiners
 Testimony in court:
Average 25 times
(IQR=7 ~ 80.75)
*For 396 examiners
Domain %
Biology, DNA 24.07
Fingerprint 14.64
QD 8.68
Toxicology 6.20
Firearm/tool
mark
5.96
Multiple 17.62
Degree %
College 42.43
Masters 38.71
Ph.D. 10.67
Gender #
Men 183
Women 219
N/A 1
Work for %
Exclusively for the
prosecution
28.29
Mostly for the
prosecution
46.40
Exclusively for the
defense
0.25
Mostly for the
defense
0.74
Country %
US 82.38
Outside US 17.62
Laboratory
size
%
21+
employees
57.57
Under 5 8.44
Alone 6.95

17. "In recent years, there has been some debate over whether forensic examiners are subconsciously influenced by
prior beliefs and expectations formed on the basis of contextual information (e.g., a detective’s opinion, evidence
from other forensic domains, a suspect’s criminal history, a confession, an eyewitness) that is irrelevant to the
forensic samples they are evaluating. This phenomenon has been referred to as cognitive bias.”
Did forensic scientists recognize that they
can cognitively bias themselves?

21. • Examiners should work “linear” rather than “circular”
 Initially examining the evidence from the crime scene
 Documenting their findings before making comparisons against a target
• Blind test under shielding various prior information
 Forensic examiners must be isolated from undue influences such as direct contact with the investigating
officer, the victims and their families, and other irrelevant information
• The verification of forensic decisions
 A more controlled process in which blind and double-blind procedures are used whenever possible
• Results from AI(Artificial Intelligence) or computerized system : the order of entries should be
randomized
• Forensic science education and certification to include training in basic psychology that is relevant
to forensic work
 Perception, judgment and decision making, and social influence
How to reduce bias: proposed reforms

22. Efforts for
Quantitative
Representation
in Forensic Science

23. Efforts to Improve the Reliability
• After the 2009 NAS Report,
systematic research conducted
by forensic science institution
• Many efforts for quantification
of forensic result : developing
criteria
• Many studies on cognitive
psychology problems related
to probabilistic expression of
quantification

24. Efforts to Evaluate the Forensic Result
Likelihood Ratio (LR)
Verbal Expression
ENFSI SWGDAM
1 Do not support Uninformative
2 ~ 10 Weak support
Limited support
10 ~ 100 Moderate support
100 ~ 1,000 Moderately strong support
Moderate support
1,000 ~ 10,000 Strong support
10,000 ~ 1,000,000 Very strong support Strong support
1,000,000 and above Extremely strong support Very strong support

25. Efforts to Evaluate the Forensic Result

26. Efforts in NFS
for
Audio Forensics

27. Principle of Voice Generation
Vocal Tract
Resonance
Vocal Folds
Resonance
Vocal Folds
Voice

28. Vocal Folds
http://www.voicedoctorla.com/voice-disorders/vocal-nodules-nodes/

29. Spectrogram, Voiceprint
The 3rd Formant
The 2nd Formant
The 1st Formant

30. Real Cases
• □: “이해를 했어요?” ( “Understand?” in Korean)
#1
#2
The 3rd Formant
The 2nd Formant
The 1st Formant

31. The Limitation of Voiceprint Analysis
Ambiguity in voiceprint analysis
• M. A. Young et al., Effects of context on talker identification, JASA 42, 1250-
1254(1967).
• K. N. Stevens et al., Speaker authentication and identification: A comparison of
spectrographic and auditory presentations of speech material, JASA 44, 1596-
1607(1987).
• O. Tosi et al., Experiment on voice identification, JASA 51, 2030-2043 (1972).
• B. E. Koenig, Spectrographic voice identification: A forensic survey, JASA 79, 288-
2090 (1986).
Test results according to the voiced information and data acquisition conditions
- Specified paragraph
- Listen with voiceprint
Errors among the forensic examiners

32. Voice Comparison Based on
Voiceprint Analysis
• Method
 Structural characteristics
 Habit of vocalization
 Linguistic characteristics
 Inter speaker variation is much larger than intra speaker variation
• Limitation
• The length of speech data at least 7s
• Language, vocalization condition, text dependency
• Results variance by examiner’s proficiency
• Limitation to numerical representation of error rate and accuracy
• Limitation to improve promptness, vulnerable to large data analysis

33. How Could We Deal with Following Problems?
Images
[1] 「名探偵コナン」の原作公式サイト, https://www.conan-portal.com/#newsContent
[2] 조선일보 연예, http://news.chosun.com/site/data/html_dir/2018/09/06/2018090603647.html
Video Clip
[3] Youtube, https://www.youtube.com/watch?v=CgX4uJSj00Y, Mission Impossible 3, 2006.

35. The Blind Test Results in NFS for Audio
Forensics
• Result of Forensic Examiners: 3 persons
• False Rejection Rate, FRR
• False Acceptance Rate, FAR
Total (200
questions)
Acceptance of decision difficulty
Non-acceptance of decision
difficulty
Correct-answer
rate
Wrong-answer
rate
Decision
difficulty
Correct-answer
rate
Wrong-answer
rate
1-person test 56.00 6.33 37.67 90.44 9.56
2-person test 43,67 1.50 54.83 96.65 3.35
3-person
unanimous
agreement
39.00 0.50 60.50 98.73 1.27

36. Result Analysis
• Characteristics of the forensic examiner’s blind test
 FRR > FAR
 FAR → 0
 The post-test exchange of opinions
influenced their decision
1-person test
3-person unanimous
agreement
2-person test
Total

37. The Blind Test Results of Laypersons
• Auditory test by laypersons
Total
Total
Age
Gender
20s
30s
40s
50s
M
F
Correct Incorrect N/A
Acceptance of decision difficulty No acceptance of
decision difficulty
Correct Incorrect

38. Result Anaysis
For forensic examiner(FE)’s unanimous
agreement cases
 FRR > FAR for laypersons
• Characteristics of the laypersons’ blind test
Total
M
F
Total
FE’sunanimousQ
M
F
Total

39. Machine Learning Based Statistical
Voice Comparison
• Low dependency on language, vocalization condition, text
• Less variance among examiners
• Possible to numerical representation of error rate and accuracy
• Possible to large data analysis

40. SIS II Quality Test
• 1:1 Comparison
30 40 50 60 70 80 90 100
10
20
30
40
50
60
70
80
90
100
Correct Rate
w/ Pitch
w/o Pitch
SF-GMM
Response Rate
w/ Pitch
w/o Pitch
SF-GMM
Rate
[%]
Data Acceptance Cutoff [%]

41. NFS Own Developed System
 For 80 samples nxn comparison : 1.5hrs (in 2019)

42. Decision Level
Under 10% error rate
Possible to suggest for the
decision difficulty in
spectrogram-based analysis
Possible to improve by
following developed algorithm
89.1 92.2

43. Voice-Phishing Fraud Group
• In 2017, a voice-phishing fraud
group was arrested
With large numbers of voice data
By using machine-learning based
system
Revealing another guilt

44. Terima kasih!