The document summarizes a study that used a hybrid lens model to examine how the quality of an automated decision aid affects human judgment performance in identifying aircraft. Participants completed scenarios where they judged aircraft types based on raw information with or without an aid. The aid's validity, reliability, and understandability were varied. The model measured the achievement and cognitive processes of participants unaided, aided, and the aid alone. It provided measures to compare the correlated judgments and determine how much participants relied on the aid versus their own cognition. The results helped identify how aid quality impacts human judgment under uncertainty.

1. Ecological Approach to Model Human Judgment
Performance under Uncertain Environment with
Autonomous Aids
TECHLAV Annual Meeting, July 24-25, 2016
Marcia Nealy (MS student)
| Dr. Younho Seong (NCAT) | Dr. Joseph Stephens (NCAT)
Hybrid Lens Model of Operator Judgment with
Autonomous Aids
Based on the Lens model with its extension, a Hybrid Lens
model was developed to precisely identify the effect of
automated decision aids on operators’ judgment
performance, depending on the quality of decision aids. The
model identifies :
• Human operators’ sole judgment performance
• Decision aids’ quality of producing estimates – Decision
aids’ competence
• Effect of decision aids’ on judgment performance
• Correspondence between the decision aid and human
operator
Introduction
• Simulated Aircraft Identification Task with
Automated Decision Aid
– Participants received raw information (radar, speed, altitude)
and output from ADA
• Varied Properties of the Aid
– Aid Validity (how well the aid predicts the aircraft type)
– Aid Reliability (how consistently the aid makes decisions)
– “Meta-information” – whether the aid reports on its own
performance
• Participants Completed Multiple Scenarios,
Over 2 Days
– Day 1 – No Aid | Day 2 – Aid, under different conditions
• High(+) vs. Low (-) Validity (V), Reliability (R), Meta Information/ Understandability (U)
Technical Experimentation
Methods Results
• Brunswik (1955), Hammond, Stewart,
Brehmer, & Steinmann, (1975)
• Achievement – The Correspondence
Between Judgments, and the Environment
to be Judged, Based on Available
information (the cues)
– Example: Judge if an aircraft is hostile or friendly based on
known information about the aircraft – achievement measures
how well your judgments correspond to the true nature of the
aircraft
• Multiple Regression Based Model
Formulation Provides Measures of:
– Achievement (ra)
– Degree to which judgment policies of the person match a
normative model of the environment (G)
– Degree to which the environment is well modeled with a
linear model (Re)
– Degree to which people make consistent judgments (Rs)
• Apply This Modeling Framework to:
– Human Operator (HO) making judgments unaided, based on a
set of cues
– Automated Decision Aid (ADA) making judgments, based on a
set of cues
– Human Operator making judgments in conjunction with the
decision aid (HO-ADA), where there is an additional cue – the
output of the ADA
– Achievement of these three systems, as well as the
correspondence (or reliance/use) of the HO and the ADA, and
the difference between the unaided HO, and aided HO, can
then be determined
Research Thrust Area: 3-7
Acknowledgement
This research is supported by Air Force Research Laboratory
and OSD under agreement number FA8750-15-2-0116, and also
Army Research Laboratory through the second author.
Achievement ra
YS
Judgments
YE
Criterion
Knowledge G
Un-modeled Agreement C
Cognitive
Control,
RS
Environmental
Predictability,
RE
Cue1
Cue2
Cue3
Cue4
Δ
Residuals
β1E
β2E
β3E
β4E
ŶE
Linear Model
Output
Δ
Residuals
β1S
β2S
β3S
β4S
ŶS
Linear Model
Output
YENV
X1
YADA
X2
X3
X4
Environmental
State
Automated
Decision
Aid
CUES
Validity Understand
-ability
Reliability