1. Subjective Integration of
Probabilistic Information from
Description and Experience
Yaron Shlomi, PhD
University of Maryland
September,
2012
Contact info: ys.work.01@gmail.com

2. Outline
n Background
n Research questions
n Importance
n Method
n Results
q Standard analyses
q Model-based analyses
n Discussion
n Open questions
2

3. Integrating Description and Experience
n Information about the prevalence of a particular medical
condition is often provided by two sources:
1. Experience: information obtained directly (e.g., from your
clinical practice)
2. Description: information observed, abstracted and
communicated from other sources (e.g., journal article)
n How do people (e.g., physicians) estimate prevalence
rates obtained from description and experience?
n More generally, how do people integrate probabilistic
information obtained from the two sources?
3

4. Description and experience:
Same or different?
n The nature of the integration depends on how
information from description and experienced is
processed.
n Differences in information source might lead to
differences in information processing.
q If so, the processing differences between description
and experience will affect how information from the
two sources is integrated.
4

5. Description and experience:
Same or different?
n The description-experience gap: Described and
experienced information about risky options yield
different patterns of choices (e.g., Hertwig & Erev, 2009).
n The description-experience gap lends support to the
hypothesis that the two sources are not processed in an
identical manner.
5

6. Description and experience:
Same or different?
n The description-experience gap might be due to
differences in presentation format (Hau, Pleskac, & Hertwig, 2010):
q Description typically provides information using
percentage or probability formats.
q Experience provides information about relative
frequencies
n Judgments based on relative frequencies are more
accurate than those based on percentages or
probabilities (Gigerenzer & Hoffrage, 1995).
6

7. Research questions
1. To what extent does the subjective integration of
description and experience deviate from optimal
standards of integration?
2. What role, if any, does the presentation format play in
the integration?
3. Are description and experience treated as equivalent
sources of information?
4. Is the nature of the integration related to the trust in the
source of the description?
5. What are the processes involved in subjective
integration of description and experience?
7

8. Importance
1. Integration of description and experience informs
potentially consequential judgments and choices.
2. Previous research on integrating the two sources is
scarce and focused on choice tasks. Conclusions may
not be generalizable to judgment tasks.
3. The task developed for this investigation could be
modified to assess integration in more complex
scenarios.
8

9. Methodology
n Participants’ task was to estimate the percentage red
chips in a bag containing red and blue chips.
n Participants received information about two independent
samples of 13 chips each from the bag.
q One sample was experienced; the other was described
9

10. Operational definitions
Description sample: Experience sample:
Summary of a Trial-by-trial
sample of chips. sampling of
blue and red
e.g., Mr. Rick chips.
sampled 13 chips.
62% of the chips in
Mr. Rick’s sample
were red.
Judgment:
Estimate percentage
of red chips in the bag.
10

11. Manipulating Presentation Format
n The numerical format of the description was manipulated
between-participants .
Format Illustrative description
Percentage Mr. Rick sampled 13 chips.
(n = 51) 62% of the chips were red.
Relative frequency Mr. Rick sampled 13 chips.
(n = 50) 8 chips were red.
11

12. Stimuli
•
16 pairs of samples obtained from •
16 additional pairs.
description and experience •
In these pairs, the experienced
•
In all of these pairs, the experienced sample is less extreme than the
sample is more extreme than the described one (e.g., 80% vs. 90%).
described one (e.g., 90% vs. 80%).
•
Does the integration depend on the
assignment of the two samples to the
two sources?
•
Are description and experience
interchangeable sources of information?
12

13. Results
n Standard analyses
n Model-based analyses
13

14. Optimal judgments
n The quality of the subjective integration is
measured by comparing the participants’
judgments to a set of optimal judgments.
n The optimal judgment is defined as the average
of the description- and experienced- based
sample proportions.
q The definition of this judgment as optimal assumes
that the two samples are obtained independently and
are equally reliable (i.e., the sample ns are equal).
14

15. Illustrating deviations from optimality
n The observed judgments
are compared to the
optimal judgments.
n If optimal, the observed
judgments would lie on
the identity line.
n If suboptimal, the
observed judgments are
either more extreme or
less extreme than the
optimal judgments.
q Equivalently, the observed
judgments are anti- 15
regressive or regressive.

16. Deviations from optimality
Percentage format
n The direction of the
deviations in the percentage
format depends on the
extremity of the experienced
sample relative to the
described one.
16

17. Deviations from optimality
n The relative frequency format leads to more optimal
judgments than the percentage format.
Percentage Relative
frequency
17

18. Deviations from optimality
•
The extremity of the judgments covaried with the extremity of the
experienced sample.
•
The relative frequency format attenuates the suboptimality of the
judgments.
Observed more
extreme than optimal
Observed less
extreme than optimal
•
Format, F(1, 99) = 1.31, p = .26
•
Assignment , F(1, 99) = 26.97, p<.001
18
•
Interaction, F(1, 99) = 7.79, p < .01

19. Integration, Format and Trust
q After completing 32 trials participants rated the
following statement,
q “I trusted Mr. Rick to provide reliable
information about the bag of chips.”
q Participants responded by marking a 5-point
scale labeled with “Completely disagree”,
“Somewhat disagree”, “Neutral”, “Somewhat
agree” and “Completely agree”.

20. Integration, Format and Trust
n Participants perceived the descriptions (i.e., Mr. Rick’s
estimates) as more trustworthy when presented as
relative frequencies compared to percentages.
“I trusted Mr. Rick to provide reliable information
about the bag of chips.”
A Kolmogorov-Smirnov test confirms
that the trust ratings differ as a
function of the presentation format
(p < .05)
20

21. Standard analyses:
Summary and reflection
n Optimality of the judgments is affected by the sample
assignment and its interaction with the presentation
format.
n Perceived trust is related to the presentation format.
n What processes underlie these findings?
21

22. Modeling subjective integration
n What constructs link the participants’ overt judgments to the
experienced and described samples?
n Hypotheses about the relevant constructs are represented
by mathematical models.
n The model fits lead to measurements of the pertinent
constructs.
q in this research, the model fits yield measurements of the
subjective sensitivity to the two information sources.
22

23. Model assumptions
n The model is motivated by two assumptions:
1. Scaling • The subjective description- and experience-
based samples are not necessarily identical
to their objective counterparts.
•The relationship between the subjective and
the objective samples can be approximated by
a suitable equation.
2. Averaging The subjective representations of the two
samples are integrated by an averaging-like
process.
23

24. Overview of the model derivation
•
The model yields a predicted judgment for any pair of
samples.
•
The predicted response is derived by combining the scaling
and averaging assumptions.
•
The model relates the participants’ judgments to the
descriptions and experiences they received.
24

25. Model derivation details
Scaling p D = κ D PD + 50(1 − κ D ) P Actual sample
p E = κ E PE + 50(1 − κ E ) p Subjective sample
κ Subjective sensitivity
Averaging ˆ
R = .5( p D + p E ) ˆ
R Predicted judgment
Model- ˆ
R = α D PD + α E PE + 50[1 − (α D + α E )] α i = .5κ i
predicted
judgment
25

26. Model Parameter Estimation
n Model parameters (i.e., the two αs) are estimated
separately for each participant.
n Parameters are estimated using Maximum Likelihood
Estimation procedures.
q i.e., loosely speaking, the goal of the procedure is to
find the parameter values that maximize the likelihood
of observing the responses,
e.g., P(R | model params).
26

27. Error theory
•
The application of the model to the data requires an error theory.
•
Observed response =
model predicted response + error,
ˆ
R = R+ε
•
We assumed ε to be normally distributed,
●
with mean θ (theta),
●
and standard deviation ς (sigma).
27

28. Log likelihood expression
32
L= ∑ ln[ f (ε )]
t =1
t
28

29. Model fitting
•
The model requires estimation of four parameters
[α D , α E , ς ,θ ]
Starting values Permissible values
α D = .5 | α D |≤ 1
α E = .5 | α E |≤ 1
ς =0 | ς |< 90
θ = 12 .01 < θ < 40
•
Fits of nested models follow similar procedure
29

30. Model predictions
•
The observed and model-predicted Observed
judgments of an illustrative
participant are plotted as a function
of the optimal judgments.
•
The model-predicted judgments
capture the relationship between the
extremeness of the experienced
sample and the extremeness of the
participant’s judgments.
Predicted
30

31. Model results
The sensitivity to description was
substantially lower than the optimal
(i.e., .5).
The sensitivity to experience was
slightly higher than the optimal (.5).
•
The relative frequency format minimizes
differences between processing MANOVA with the two αs as the dependent
description and experience. variables and the description format as its
factor :
Format, F(2, 98) = 5.7, p < .01.
Format effects on αD and αE are F(1, 99) =
11.53, p < .01 and (1, 99) = 1.76, p = .19,
respectively. 31

32. Hypotheses about the integration
n The subjective sensitivities to description and experience
might be related to each other.
q The model, as presented to this point, does not specify form of the
relationship (if any).
q The results, however, suggest that the sensitivities are inversely
related to each other (i.e., the sensitivities trade-off).
32

33. Hypotheses about Integration
n How, if at all, do the sensitivities to description and
experience relate to each other?
1. the sensitivities trade-off
2. the sensitivities are equal to each other
3. the sensitivities are equal to each other and optimal
(i.e., both equal to .5)
n The model yields tests of these hypotheses at the
individual-participant level.
33

34. Hypothesis Testing: Restricted Models
Hypothesis Putative
n Hypotheses about the
integration process are relationship
represented by postulating
specific relationships between Unrelated
the model parameters.
n The hypotheses are tested by Tradeoff
comparing model fits (e.g., G2 αD +αE =1
tests and related procedures) Equal sensitivity
α D = α E ≠ .5
Optimal α D = α E = .5
34

35. Model comparisons: Key finding
• The best-fitting model Hypothesis / Putative
depended on the description
format. Model relationship
• In the percentage format, the
tradeoff model outperformed the Unrelated
equal-sensitivity model.
• In the relative frequency format, Tradeoff
the equal–sensitivity model αD +αE =1
outperformed the tradeoff model.
Equal sensitivity
α D = α E ≠ .5
Optimal α D = α E = .5
35

36. Trust and Subjective Integration
•
The participants’ trust in the
description was related to their
sensitivities to description and
experience (i.e., as inferred from
the model)
•
Specifically, the parameter
estimates show that more trust
was associated with more
comparable sensitivities to
description and experience.
“I trusted Mr. Rick to provide reliable
information about the bag of chips.”
36

37. Summary of the findings
Optimality of the judgments
Standard analyses The observed judgments deviated
from the optimal judgments in the
direction of the experienced sample.
Added value of The model fits show that the
using the model sensitivity to description was less
optimal than the sensitivity to
experience .
37

38. Summary of the findings
Effects of presentation format
Standard analyses The format did not lead to statistically
significant effects.
Added value of Manipulating the description format of
using the model the affects the sensitivity to
description.
The sensitivity to experience was
unaffected.
38

39. Summary of the findings
Effect of assigning the extreme sample
to experience vs. description
Standard analyses Judgments were affected by the
information assignment.
The extremity of the judgments
covaried with the extremity of the
experienced sample relative to the
described one.
Added value of The algebra of the model shows that
using the model the assignment effect is expected if
the sensitivites are unequal and the
tradeoff is imperfect.
39

40. Summary of the findings
Psychological process
Standard analyses Standard analyses are mute regarding
the underlying psychological process.
Added value of The model fits suggest that the
using the model presentation format determines the
form of processing (equal sensitivity,
tradeoff).
40

41. Summary of the findings
Trust
Standard analyses Trust in the description varied as a
function of the description format (i.e.,
percentage vs. relative frequency).
Added value of The estimated model parameters are
using the model correlated with the participants’ trust
ratings.
41

42. Brief Discussion
1. Optimality The finding of suboptimal integration is
important for motivating theory, for
anticipating the quality of the judgments in
real-world tasks, and for designing decision
aids.
2. Format The similarity of the processes invoked by
description and experience depends on the
source’s presentation format (c.f. Gottlieb, Weiss, &
Chapman, 2007)
3. Trust Participants’ trust ratings might be informed
by their assessments of how fluently they
process information in a particular format.
42

43. Brief Discussion
4 Information The assignment effects support the
assignment hypothesis that description and
experience are not processed in a similar
way.
5 Process Source-related differences in the
sensitivity to outcome probabilities might
be the dominant factor in the description-
experience gap in risky choice.
43

44. Discussion: Benefits of modeling
n The model provides a framework for testing hypotheses
about pertinent constructs.
q Without a model, these hypotheses are very difficult to
test.
n The model removes noise in the data, rendering the
patterns in the subject’s responses easier to detect.
n Model parameters (as indicators of underlying process)
provide useful information for understanding individual
differences.
44

45. Open questions
1. How well does the task mimic people’s information
integration in more complex situations?
q e.g., tasks that involve sources that differ in their
trustworthiness, multidimensional information, …
2. How do people allocate their processing resources (e.g.,
attention) between description and experience?
3. Are the representational differences between description
and experience sufficient to account for the description-
experience gap in risky choice?
4. What is the role of trust in the descriptions in the
description-experience gap in risky choice?
45

46. Acknowledgments
University of Maryland
Faculty and Postdocs Research Assistants
Thomas Wallsten, Chair (PSYC) Joshua Boker
Thomas Carlson Ezra Geis
Michael Dougherty Leda Kaveh
Rebecca Hamilton Marissa Lewis
Carl Lejuez
Stephanie Odenheimer
Cheri Ostroff
Lauren Spicer
Hsuchi Ting
Herschel Lisette Sy
Kimberly White
46