1. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 1
Perception, Comparison,
and Models
for Uncertainty
Michael Gleicher
Department of Computer Sciences
University of Wisconsin - Madison
Slides from an invited talk at
EuroRV3, 4 June 2018
Not all slides were shown
Photographs of other peoples’
slides used with permission
2. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 2
3. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 3
Acknowledgements
Students – past and present
Michael Correll, Eric Alexander, Danielle Szafir, Alper Sarikaya, …
Collaborators – vis and domain science
Steve Franconeri, Remco Chang, Kristi Potter, Michael Witmore …
Those who invited me…
Who forced me to re-think some old stuff Slides from an invited talk at
EuroRV3, 4 June 2018
Not all slides were shown
Photographs of other peoples’
slides used with permission
4. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 4
Summary
Models (or lack of) are central to uncertainty
but sometimes we hide them or forget them – for better/worse
Communicating models may be a core problem
at least it provides a way to think about things
Perception and design can help communicate models
but we haven’t been thinking about that enough
Thinking this way may be useful for many problems
but I haven’t figured it out yet
5. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 5
Outline
Framework –uncertainty issues as model communication
Examples of two paths to model communication
standard path (error bars)
non-standard (?) path (implicit uncertainty)
Reconnecting to the broader problem
6. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 6
Connected to others today…
Slides from an invited talk at
EuroRV3, 4 June 2018
Not all slides were shown
Photographs of other peoples’
slides used with permission
7. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 7
From this morning’s opening
talk…
Reproducibility and verifyability and …
1. of research results…
2. providing tools for others to do it for their work…
Uncertainty
(I was asked to talk about this)
8. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 8
From this morning’s keynote talk
Mental model is part of things
9. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 9
Things I will challenge
We need to quantify
uncertainty to work with?
Thinking in terms of errors?
Uncertainty = true randomness
10. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 10
One form of variation can obscure another
But…
One person’s noise is another’s data
spelling obscures ideas
or
observe the development of spelling
rotting pages obscure content
or
preserved books tell us what was valued
Warning:
Slide out of Context
11. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 11
Where we really differ…
Formalism/Model is end User Task is end
12. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 12
The questions afterwards…
What about machine learning models?
“They” do have models – they just don’t understand them
Do users really understand these quantitative models?
13. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 13
A Framework
Complex and/or
Unmeasurable Process
Models Observations /
Data
Sample from
Model
Simpler/Emprical
Model
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
Model
Visualizaton
Multi-Object
Visualization
14. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 14
What is uncertainty?
15. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 15
What is uncertainty?
Keynote
Hege citing Adrienko
First invited talk
Gschwandtner citing Peng
16. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 16
What is uncertainty?
Our information doesn’t fully specify the outcome
• Limited model (e.g., unmodeled factors)
• Limited measurements/observations
• Truly random process
Variance – not explained by inputs
Variety of possible outcomes
17. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 17
So what?
There are many possible
outcomes (predictive/explanatory)
states/configurations (description)
We may understand which ones are more likely
18. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 18
Why is this a problem?
Users need to do something – despite uncertainty!
Make decisions in the face of uncertainty (predict)
Build a better model that accounts for more variance (explain)
Describe (concisely) the set of observations/outcomes (describe)
or describe the underlying state/configuration
19. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 19
Shmueli, G. (2010). To Explain
or to Predict? Statistical
Science, 25(3), 289–310.
20. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 20
The three statistical tasks
Predict
What will the outcome be in an unseen situation?
performance predict: how likely is the outcome my model predicts?
Explain (statistical)
Does my theory explain what I observe? (or is it a coincidence)
explain (Vis Analytics) – help me develop causal theories
explain (ML/HCI) – help me understand a model’s prediction
Describe
Represent the data in a concise form
21. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 21
A Framework
From here to there…
Complex and/or
Unmeasurable Process
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
22. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 22
Where does
visualization fit
in?
Complex and/or
Unmeasurable Process
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
23. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 23
A Framework
WARNING:
Assertion– without evidence
Viewer builds a model
Viewer uses the model
Maybe wrong – but useful
Complex and/or
Unmeasurable Process
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
24. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 24
Central Premise
People build internal models
how do we help them do that?
People use internal models
25. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 25
All models are wrong,
Some are useful
George Box
26. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 26
Mental Model
vs.
Math Model
Models of uncertainty
Models with uncertainty
Complex and/or
Unmeasurable Process
Models Observations /
Data
Sample from
Model
Simpler/Emprical
Model
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
Model
Visualizaton
Multi-Object
Visualization
27. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 27
Sample from
Model
Simpler/Emprical
Model
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
Model
Visualizaton
Multi-Object
Visualization
Where do
models come
from?
Designed Models
Empirical Models
(fit from data)
(machine learning)
Complex and/or
Unmeasurable Process
Observations /
Data
Models
28. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 28
A thought…
Building the mental model
is the key task?
Underlying process
Uncertainty in data
Complex and/or
Unmeasurable Process
Models Observations /
Data
Sample from
Model
Simpler/Emprical
Model
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
Model
Visualizaton
Multi-Object
Visualization
29. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 29
The Standard Path
Depicting Models of Uncertainty
30. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 30
Visualizing
Models of
Uncertainty
Complex and/or
Unmeasurable Process
Models Observations /
Data
Sample from
Model
Simpler/Emprical
Model
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
Model
Visualizaton
Multi-Object
Visualization
31. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 31
The simplest case…
A scalar value – with measurements
75 +/- 5
This is a model – expectation and variance
We assume the 2 numbers is a sufficient description
the model is appropriately selected
Warning:
This is ambiguous!
• Standard error?
• Standard deviation?
• Range (max/min)?
• 95% confidence interval?
32. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 32
There is a standard design…
0
10
20
30
40
50
60
70
80
90
100
Placebo Treatment
33. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 33
Correll, M., & Gleicher, M. (2014).
Error Bars Considered Harmful: Exploring Alternate
Encodings for Mean and Error. IEEE Transactions
on Visualization and Computer Graphics, 20 (12),
2142–2151.
34. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 34
There are problems…
Ambiguous
All or nothing
Asymmetric
Not salient
0
10
20
30
40
50
60
70
80
90
100
Placebo Treatment
35. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 35
Ambiguous: Label your charts!
We should know better!
For those that were labeled…
Standard error
95% t confidence interval
Range
1.5 x interquartile range
Standard deviation
80% t confidence interval
InfoVis 2010-2013
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Labeled
Unlabeled
36. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 36
Perceptual problems…
Ambiguous
All or nothing
Asymmetric
Not salient
0
10
20
30
40
50
60
70
80
90
100
Placebo Treatment
37. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 37
Alternative Designs
More salience for uncertainty
Symmetric (avoid within-the-bar bias)
Gradual (not all-or-nothing)
38. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 38
What is better?
Care about task performance
not accuracy
Viewer confidence correlates
with statistical answer
39. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 39
Similar experiments this morning
great focus on real tasks in evaluation!
40. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 40
One-sided test
How likely (or how
surprising) do you
think the red
potential outcome
is, given the poll?
41. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 41
Two sided test
If forced to guess, which city do
you predict will get more snow?
How confident are you?
How one-sided do you predict?
42. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 42
Judgments correlate with
uncertainty
43. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 43
Messages from the paper
What people remember
Error bars have perceptual problems – consider other designs
The bigger messages
Label your models!
Assess performance based on tasks
People can actually do the tasks – no matter which design
44. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 44
Main Messages
Models matter – communicate them correctly!
Perceptual issues matter – design correctly!
People get the right answers – qualitatively
On a limb in 2018:
Maybe what matters is building the right mental model?
45. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 45
Alternative Designs
Hypothesis (totally untested):
These designs lead to similar mental models
The details of the designs don’t matter much
46. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 46
You can make pictures of models
But you need the model to make a picture of…
And tricky things come up in designing them…
It’s not clear what mental model gets build from the picture
The mental model might be what matters
47. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 47
Really out on a limb…
Hypothesis (totally untested):
These designs lead to similar mental models
The details of the designs don’t matter much
Hypothesis (total speculation):
People are “good” at using their mental models
Exactly what model they have doesn’t matter as much
48. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 48
Can my theory explain her data?
49. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 49
An Alternate Pathway?
50. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 50
Complex and/or
Unmeasurable Process
Models Observations /
Data
Sample from
Model
Simpler/Emprical
Model
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
Model
Visualizaton
Multi-Object
Visualization
Avoid the
model?
51. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 51
Correll, M., & Gleicher, M. (2015). Implicit
Uncertainty Visualization: Aligning Perception and
Statistics. In Proceedings of the 2015 Workshop on
Visualization for Decision Making Under
Uncertainty.
52. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 52
Thrice rejected paper warning!
53. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 53
A little background…
54. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 54
Visual Aggregation
(Short version)
People can estimate various statistical properties
including mean and “variance”
Different designs can help / hurt
55. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 55
Which Color Point is Higher
on Average?
Gleicher, M., Correll, M., Nothelfer, C. and Franconeri, C. “Perception of
Average Value in Multiclass Scatterplots.” InfoVis 2013
56. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 56
How did you do that?
57. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 57
Larger differences gives
better performance
More points do not hurt
performance
Stronger cues (color)
outperform weaker ones
Redundant cues do not help
performance
Conflicting cues do not hurt
performance
Distractor class does not
hurt performance
Key Results
Gleicher, M., Correll, M., Nothelfer, C. and Franconeri, C. “Perception of
Average Value in Multiclass Scatterplots.” InfoVis 2013
58. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 58
Visual Aggregation
People can extract summary statistics
Which Ones?
Efficiently?
Accurately?
How?
What can we do with it?
Why should we use it?
59. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 59
Visual Aggregation
Empirical Understanding
Averages in Time Series
Correll, et al. CHI 2012
Tagged Text
Correll, et al. CHI 2013
Scatterplot Averages
Gleicher, et al. InfoVis 2013
Other statistics in Time Series
Albers, et al. CHI 2014
Practical Application
Sequence Surveyor (Genetics)
Albers, et al. InfoVis 2011
LayerCake (Virus mutations)
Correll, et al. BioVis 2011
Molecular Surface Experiments
Sarikaya, et al. EuroVis 2014
Decision Making
Correll, and Gleicher (2015)
Correll et al.
60. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 60
Szafir, D. A., Haroz, S., Gleicher, M., &
Franconeri, S. (2016).
Four types of ensemble coding in
data visualizations. Journal of Vision,
16(5), 11.
https://doi.org/10.1167/16.5.11
61. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 61
Design for Aggregation
Different designs
Different tasks
Some are better
than others
62. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 62
Time Series?
Correll, Albers, Franconeri, Gleicher. Comparing Averages in Time
Series Data. CHI 2012.
63. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 63
Which month has the highest
average?
64. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 64
65. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 65
66. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 66
67. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 67
Conditions
Linegraphs:
Regular or 1D permuted
Colorfields:
Regular or woven
68. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 68
Results
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7 8 9 10
Accuracy
Difference Between Averages (d)
Woven Colorfield
Colorfield
Linegraph
Permuted Linegraph
69. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 69
What besides averages?
Albers, Correll, Gleicher. Task-Driven Evaluation of Aggregation in Time Series
Visualization. CHI 2014.
70. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 70
Things You Might Care About
1. Maxima: Which month had the day with the highest
sales for the year?
2. Minima: Which month had the day with the lowest
sales for the year?
3. Range: Which month had the largest range of values?
4. Average: Which month had the highest average sales
for the year?
5. Spread: Look at the average sales from each month.
Which month had the sales which were the most
spread out from their monthly average?
6. Outliers: Which month had the most unusual (outlier)
sales days?
71. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 71
Position
Line Graph Box Plot
Composite Graph Stock Chart
72. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 72
Color
Colorfield Woven
Event Striping Color Stock Chart
73. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 73
Results
74. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 74
Visual Aggregation
(Short version)
People can estimate various statistical properties
including mean and “variance”
Different designs can help / hurt
75. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 75
What does this have to do with
uncertainty?
76. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 76
Why Visual Aggregation?
Why not just give them the answer?
1. You may not know what the viewer wants
2. Some aggregate properties might be complicated
3. You can’t show all properties
4. It gives the viewer more information
5. Doing “work” might force them to think about things
6. Uncertainty of perception maps to uncertainty in data
77. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 77
Implicit Uncertainty
2015 Version
Error Bars (mean+variance=decision)
+
Visual Aggregation (see mean&variance)
=
Uncertainty without the (explicit) model!
2018 Version?
78. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 78
Complex and/or
Unmeasurable Process
Models Observations /
Data
Sample from
Model
Simpler/Emprical
Model
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
Model
Visualizaton
Multi-Object
Visualization
Avoid the
model?
Viewer builds the model
themselves!
Make decisions based on
internal model
79. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 79
Do you get the same answer?
Implicit (show data)
Explicit (show model)
Annotative (show both)
80. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 80
81. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 81
Viewers get similar answers
Judgments are qualitative
Similar in magnitude
Correctly correlated with actual
uncertainty
82. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 82
Prediction tasks
Estimate the next point (in Anscombe’s quartet)
Complicated modeling question
throw out outliers?
what kind of curve to fit?
83. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 83
84. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 84
Why reviewers (correctly) hated it
A few toy examples
Unclear what guidance is being given
(when to use it, what designs to use, …)
Expository Issues:
Unclear how this is different from standard statistical vis
85. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 85
Why reviewers (should) hate it
May break in the small
Qualitiatively right in easy cases
Very simple models
Perceptual inaccuracies
Perceptual biases
Cognitive biases
Scales Horribly?
More data?
More complex items?
More complex decisions?
Other design properties?
86. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 86
Complex and/or
Unmeasurable Process
Models Observations /
Data
Sample from
Model
Simpler/Emprical
Model
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
Model
Visualizaton
Multi-Object
Visualization
Avoid the
model?
87. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 87
My 2018 (post-hoc) explanation
People build an internal model
People use this internal model
88. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 88
Complex and/or
Unmeasurable Process
Models Observations /
Data
Sample from
Model
Simpler/Emprical
Model
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
Model
Visualizaton
Multi-Object
Visualization
Directly
construct the
model from data
89. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 89
Why does this work?
People can build internal models
present the data so that people can do this
People can use internal models robustly
internal model may not be great
it just works for the specific task
90. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 90
Complex and/or
Unmeasurable Process
Models Observations /
Data
Sample from
Model
Simpler/Emprical
Model
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
Model
Visualizaton
Multi-Object
Visualization
Avoid the
model?
We didn’t us the math
model
No reasons to believe the
users’ model aligns with it
It just gets same results
91. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 91
Would you actually do this?
May break in the small
Qualitiatively right in easy cases
Very simple models
Perceptual inaccuracies
Perceptual biases
Cognitive biases
Scales Horribly?
More data?
More complex items?
More complex decisions?
Other design properties?
92. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 92
What if you had no choice?
Model is too complicated to visualize
Model isn’t known
Model is too hard to select
Variance is too hard to characterize
Showing examples
93. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 93
A different thought…
How do we convey models by showing a set of examples?
Sometimes this is all we have (examples)
Sometimes the model is too complicated (generate examples)
Isn’t this (related to) ensemble visualization?
94. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 94
Model Sampling
Complex and/or
Unmeasurable Process
Models Observations /
Data
Sample from
Model
Simpler/Emprical
Model
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
Model
Visualizaton
Multi-Object
Visualization
95. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 95
Two Parts
Choose the right examples
Representative set selection
Experiment design
Convey the Examples
Visual Comparison
Ensemble Visualization
96. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 96
How do we do this?
Choose the Right Examples Convey the Examples
Gleicher, M. (2018). Considerations for Visualizing
Comparison. IEEE Transactions on Visualization
and Computer Graphics, 24(1), 413–423.
InfoVis 2017.
97. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 97
A new path to model usability?
Interpretable models
simplify the models so they can be understood
Examinable models
look inside the models and hope you understand
Instance-based explanations
pick some decisions and try to understand them
Experiment/Outcome Examination
look at the right input/outputs from the black box
98. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 98
A general problem:
Model Usability?
How can we work with complex models?
Machine learning
Models of complex processes
Variety of tasks
Variety of users
Variety of approaches
Interpreting an ML model is similar to
interpreting and uncertain model? Gleicher, M. (2016). A Framework for
Considering Comprehensibility in
Modeling. Big Data, 4(2), 75–88.
99. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 99
A Framework
Complex and/or
Unmeasurable Process
Models Observations /
Data
Sample from
Model
Simpler/Emprical
Model
User Mental Model
Decisions /
Predictions
Improved Models
/ Data Debugging
Understanding of
Variance
Visualization
Model
Visualizaton
Multi-Object
Visualization
100. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 100
Summary
Models (or lack of) are central to uncertainty
but sometimes we hide them or forget them – for better/worse
Communicating models may be a core problem
at least it provides a way to think about things
Perception and design can help communicate models
but we haven’t been thinking about that enough
Thinking this way may be useful for many problems
but I haven’t figured it out yet
Acknowledgments
This work is funded in part by NSF 1162037
and DARPA FA8750-17-2-0107
Michael Gleicher
Department of Computer Sciences
University of Wisconsin - Madison
Slides from an invited talk at
EuroRV3, 4 June 2018
Not all slides were shown
Photographs of other peoples’
slides used with permission
