Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Perception, Comparison, and Modelsfor Uncertainty

49 views

Published on

Slides from talk at the EuroRV3 workshop, 4 June, 2018.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Perception, Comparison, and Modelsfor Uncertainty

  1. 1. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 1 Perception, Comparison, and Models for Uncertainty Michael Gleicher Department of Computer Sciences University of Wisconsin - Madison Slides from an invited talk at EuroRV3, 4 June 2018 Not all slides were shown Photographs of other peoples’ slides used with permission
  2. 2. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 2
  3. 3. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 3 Acknowledgements Students – past and present Michael Correll, Eric Alexander, Danielle Szafir, Alper Sarikaya, … Collaborators – vis and domain science Steve Franconeri, Remco Chang, Kristi Potter, Michael Witmore … Those who invited me… Who forced me to re-think some old stuff Slides from an invited talk at EuroRV3, 4 June 2018 Not all slides were shown Photographs of other peoples’ slides used with permission
  4. 4. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 4 Summary Models (or lack of) are central to uncertainty but sometimes we hide them or forget them – for better/worse Communicating models may be a core problem at least it provides a way to think about things Perception and design can help communicate models but we haven’t been thinking about that enough Thinking this way may be useful for many problems but I haven’t figured it out yet
  5. 5. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 5 Outline Framework –uncertainty issues as model communication Examples of two paths to model communication standard path (error bars) non-standard (?) path (implicit uncertainty) Reconnecting to the broader problem
  6. 6. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 6 Connected to others today… Slides from an invited talk at EuroRV3, 4 June 2018 Not all slides were shown Photographs of other peoples’ slides used with permission
  7. 7. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 7 From this morning’s opening talk… Reproducibility and verifyability and … 1. of research results… 2. providing tools for others to do it for their work… Uncertainty (I was asked to talk about this)
  8. 8. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 8 From this morning’s keynote talk Mental model is part of things
  9. 9. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 9 Things I will challenge We need to quantify uncertainty to work with? Thinking in terms of errors? Uncertainty = true randomness
  10. 10. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 10 One form of variation can obscure another But… One person’s noise is another’s data spelling obscures ideas or observe the development of spelling rotting pages obscure content or preserved books tell us what was valued Warning: Slide out of Context
  11. 11. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 11 Where we really differ… Formalism/Model is end User Task is end
  12. 12. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 12 The questions afterwards… What about machine learning models? “They” do have models – they just don’t understand them Do users really understand these quantitative models?
  13. 13. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 13 A Framework Complex and/or Unmeasurable Process Models Observations / Data Sample from Model Simpler/Emprical Model User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization Model Visualizaton Multi-Object Visualization
  14. 14. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 14 What is uncertainty?
  15. 15. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 15 What is uncertainty? Keynote Hege citing Adrienko First invited talk Gschwandtner citing Peng
  16. 16. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 16 What is uncertainty? Our information doesn’t fully specify the outcome • Limited model (e.g., unmodeled factors) • Limited measurements/observations • Truly random process Variance – not explained by inputs Variety of possible outcomes
  17. 17. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 17 So what? There are many possible outcomes (predictive/explanatory) states/configurations (description) We may understand which ones are more likely
  18. 18. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 18 Why is this a problem? Users need to do something – despite uncertainty! Make decisions in the face of uncertainty (predict) Build a better model that accounts for more variance (explain) Describe (concisely) the set of observations/outcomes (describe) or describe the underlying state/configuration
  19. 19. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 19 Shmueli, G. (2010). To Explain or to Predict? Statistical Science, 25(3), 289–310.
  20. 20. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 20 The three statistical tasks Predict What will the outcome be in an unseen situation? performance predict: how likely is the outcome my model predicts? Explain (statistical) Does my theory explain what I observe? (or is it a coincidence) explain (Vis Analytics) – help me develop causal theories explain (ML/HCI) – help me understand a model’s prediction Describe Represent the data in a concise form
  21. 21. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 21 A Framework From here to there… Complex and/or Unmeasurable Process Decisions / Predictions Improved Models / Data Debugging Understanding of Variance
  22. 22. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 22 Where does visualization fit in? Complex and/or Unmeasurable Process Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization
  23. 23. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 23 A Framework WARNING: Assertion– without evidence Viewer builds a model Viewer uses the model Maybe wrong – but useful Complex and/or Unmeasurable Process User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization
  24. 24. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 24 Central Premise People build internal models how do we help them do that? People use internal models
  25. 25. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 25 All models are wrong, Some are useful George Box
  26. 26. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 26 Mental Model vs. Math Model Models of uncertainty Models with uncertainty Complex and/or Unmeasurable Process Models Observations / Data Sample from Model Simpler/Emprical Model User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization Model Visualizaton Multi-Object Visualization
  27. 27. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 27 Sample from Model Simpler/Emprical Model User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization Model Visualizaton Multi-Object Visualization Where do models come from? Designed Models Empirical Models (fit from data) (machine learning) Complex and/or Unmeasurable Process Observations / Data Models
  28. 28. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 28 A thought… Building the mental model is the key task? Underlying process Uncertainty in data Complex and/or Unmeasurable Process Models Observations / Data Sample from Model Simpler/Emprical Model User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization Model Visualizaton Multi-Object Visualization
  29. 29. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 29 The Standard Path Depicting Models of Uncertainty
  30. 30. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 30 Visualizing Models of Uncertainty Complex and/or Unmeasurable Process Models Observations / Data Sample from Model Simpler/Emprical Model User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization Model Visualizaton Multi-Object Visualization
  31. 31. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 31 The simplest case… A scalar value – with measurements 75 +/- 5 This is a model – expectation and variance We assume the 2 numbers is a sufficient description the model is appropriately selected Warning: This is ambiguous! • Standard error? • Standard deviation? • Range (max/min)? • 95% confidence interval?
  32. 32. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 32 There is a standard design… 0 10 20 30 40 50 60 70 80 90 100 Placebo Treatment
  33. 33. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 33 Correll, M., & Gleicher, M. (2014). Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error. IEEE Transactions on Visualization and Computer Graphics, 20 (12), 2142–2151.
  34. 34. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 34 There are problems… Ambiguous All or nothing Asymmetric Not salient 0 10 20 30 40 50 60 70 80 90 100 Placebo Treatment
  35. 35. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 35 Ambiguous: Label your charts! We should know better! For those that were labeled… Standard error 95% t confidence interval Range 1.5 x interquartile range Standard deviation 80% t confidence interval InfoVis 2010-2013 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Labeled Unlabeled
  36. 36. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 36 Perceptual problems… Ambiguous All or nothing Asymmetric Not salient 0 10 20 30 40 50 60 70 80 90 100 Placebo Treatment
  37. 37. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 37 Alternative Designs More salience for uncertainty Symmetric (avoid within-the-bar bias) Gradual (not all-or-nothing)
  38. 38. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 38 What is better? Care about task performance not accuracy Viewer confidence correlates with statistical answer
  39. 39. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 39 Similar experiments this morning great focus on real tasks in evaluation!
  40. 40. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 40 One-sided test How likely (or how surprising) do you think the red potential outcome is, given the poll?
  41. 41. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 41 Two sided test If forced to guess, which city do you predict will get more snow? How confident are you? How one-sided do you predict?
  42. 42. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 42 Judgments correlate with uncertainty
  43. 43. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 43 Messages from the paper What people remember Error bars have perceptual problems – consider other designs The bigger messages Label your models! Assess performance based on tasks People can actually do the tasks – no matter which design
  44. 44. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 44 Main Messages Models matter – communicate them correctly! Perceptual issues matter – design correctly! People get the right answers – qualitatively On a limb in 2018: Maybe what matters is building the right mental model?
  45. 45. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 45 Alternative Designs Hypothesis (totally untested): These designs lead to similar mental models The details of the designs don’t matter much
  46. 46. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 46 You can make pictures of models But you need the model to make a picture of… And tricky things come up in designing them… It’s not clear what mental model gets build from the picture The mental model might be what matters
  47. 47. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 47 Really out on a limb… Hypothesis (totally untested): These designs lead to similar mental models The details of the designs don’t matter much Hypothesis (total speculation): People are “good” at using their mental models Exactly what model they have doesn’t matter as much
  48. 48. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 48 Can my theory explain her data?
  49. 49. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 49 An Alternate Pathway?
  50. 50. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 50 Complex and/or Unmeasurable Process Models Observations / Data Sample from Model Simpler/Emprical Model User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization Model Visualizaton Multi-Object Visualization Avoid the model?
  51. 51. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 51 Correll, M., & Gleicher, M. (2015). Implicit Uncertainty Visualization: Aligning Perception and Statistics. In Proceedings of the 2015 Workshop on Visualization for Decision Making Under Uncertainty.
  52. 52. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 52 Thrice rejected paper warning!
  53. 53. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 53 A little background…
  54. 54. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 54 Visual Aggregation (Short version) People can estimate various statistical properties including mean and “variance” Different designs can help / hurt
  55. 55. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 55 Which Color Point is Higher on Average? Gleicher, M., Correll, M., Nothelfer, C. and Franconeri, C. “Perception of Average Value in Multiclass Scatterplots.” InfoVis 2013
  56. 56. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 56 How did you do that?
  57. 57. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 57 Larger differences gives better performance More points do not hurt performance Stronger cues (color) outperform weaker ones Redundant cues do not help performance Conflicting cues do not hurt performance Distractor class does not hurt performance Key Results Gleicher, M., Correll, M., Nothelfer, C. and Franconeri, C. “Perception of Average Value in Multiclass Scatterplots.” InfoVis 2013
  58. 58. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 58 Visual Aggregation People can extract summary statistics Which Ones? Efficiently? Accurately? How? What can we do with it? Why should we use it?
  59. 59. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 59 Visual Aggregation Empirical Understanding Averages in Time Series Correll, et al. CHI 2012 Tagged Text Correll, et al. CHI 2013 Scatterplot Averages Gleicher, et al. InfoVis 2013 Other statistics in Time Series Albers, et al. CHI 2014 Practical Application Sequence Surveyor (Genetics) Albers, et al. InfoVis 2011 LayerCake (Virus mutations) Correll, et al. BioVis 2011 Molecular Surface Experiments Sarikaya, et al. EuroVis 2014 Decision Making Correll, and Gleicher (2015) Correll et al. 
  60. 60. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 60 Szafir, D. A., Haroz, S., Gleicher, M., & Franconeri, S. (2016). Four types of ensemble coding in data visualizations. Journal of Vision, 16(5), 11. https://doi.org/10.1167/16.5.11
  61. 61. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 61 Design for Aggregation Different designs Different tasks Some are better than others
  62. 62. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 62 Time Series? Correll, Albers, Franconeri, Gleicher. Comparing Averages in Time Series Data. CHI 2012.
  63. 63. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 63 Which month has the highest average?
  64. 64. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 64
  65. 65. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 65
  66. 66. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 66
  67. 67. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 67 Conditions Linegraphs: Regular or 1D permuted Colorfields: Regular or woven
  68. 68. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 68 Results 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 Accuracy Difference Between Averages (d) Woven Colorfield Colorfield Linegraph Permuted Linegraph
  69. 69. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 69 What besides averages? Albers, Correll, Gleicher. Task-Driven Evaluation of Aggregation in Time Series Visualization. CHI 2014.
  70. 70. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 70 Things You Might Care About 1. Maxima: Which month had the day with the highest sales for the year? 2. Minima: Which month had the day with the lowest sales for the year? 3. Range: Which month had the largest range of values? 4. Average: Which month had the highest average sales for the year? 5. Spread: Look at the average sales from each month. Which month had the sales which were the most spread out from their monthly average? 6. Outliers: Which month had the most unusual (outlier) sales days?
  71. 71. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 71 Position Line Graph Box Plot Composite Graph Stock Chart
  72. 72. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 72 Color Colorfield Woven Event Striping Color Stock Chart
  73. 73. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 73 Results
  74. 74. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 74 Visual Aggregation (Short version) People can estimate various statistical properties including mean and “variance” Different designs can help / hurt
  75. 75. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 75 What does this have to do with uncertainty?
  76. 76. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 76 Why Visual Aggregation? Why not just give them the answer? 1. You may not know what the viewer wants 2. Some aggregate properties might be complicated 3. You can’t show all properties 4. It gives the viewer more information 5. Doing “work” might force them to think about things 6. Uncertainty of perception maps to uncertainty in data
  77. 77. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 77 Implicit Uncertainty 2015 Version Error Bars (mean+variance=decision) + Visual Aggregation (see mean&variance) = Uncertainty without the (explicit) model! 2018 Version?
  78. 78. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 78 Complex and/or Unmeasurable Process Models Observations / Data Sample from Model Simpler/Emprical Model User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization Model Visualizaton Multi-Object Visualization Avoid the model? Viewer builds the model themselves! Make decisions based on internal model
  79. 79. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 79 Do you get the same answer? Implicit (show data) Explicit (show model) Annotative (show both)
  80. 80. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 80
  81. 81. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 81 Viewers get similar answers Judgments are qualitative Similar in magnitude Correctly correlated with actual uncertainty
  82. 82. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 82 Prediction tasks Estimate the next point (in Anscombe’s quartet) Complicated modeling question throw out outliers? what kind of curve to fit?
  83. 83. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 83
  84. 84. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 84 Why reviewers (correctly) hated it A few toy examples Unclear what guidance is being given (when to use it, what designs to use, …) Expository Issues: Unclear how this is different from standard statistical vis
  85. 85. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 85 Why reviewers (should) hate it May break in the small Qualitiatively right in easy cases Very simple models Perceptual inaccuracies Perceptual biases Cognitive biases Scales Horribly? More data? More complex items? More complex decisions? Other design properties?
  86. 86. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 86 Complex and/or Unmeasurable Process Models Observations / Data Sample from Model Simpler/Emprical Model User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization Model Visualizaton Multi-Object Visualization Avoid the model?
  87. 87. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 87 My 2018 (post-hoc) explanation People build an internal model People use this internal model
  88. 88. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 88 Complex and/or Unmeasurable Process Models Observations / Data Sample from Model Simpler/Emprical Model User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization Model Visualizaton Multi-Object Visualization Directly construct the model from data
  89. 89. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 89 Why does this work? People can build internal models present the data so that people can do this People can use internal models robustly internal model may not be great it just works for the specific task
  90. 90. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 90 Complex and/or Unmeasurable Process Models Observations / Data Sample from Model Simpler/Emprical Model User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization Model Visualizaton Multi-Object Visualization Avoid the model? We didn’t us the math model No reasons to believe the users’ model aligns with it It just gets same results
  91. 91. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 91 Would you actually do this? May break in the small Qualitiatively right in easy cases Very simple models Perceptual inaccuracies Perceptual biases Cognitive biases Scales Horribly? More data? More complex items? More complex decisions? Other design properties?
  92. 92. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 92 What if you had no choice? Model is too complicated to visualize Model isn’t known Model is too hard to select Variance is too hard to characterize Showing examples
  93. 93. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 93 A different thought… How do we convey models by showing a set of examples? Sometimes this is all we have (examples) Sometimes the model is too complicated (generate examples) Isn’t this (related to) ensemble visualization?
  94. 94. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 94 Model Sampling Complex and/or Unmeasurable Process Models Observations / Data Sample from Model Simpler/Emprical Model User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization Model Visualizaton Multi-Object Visualization
  95. 95. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 95 Two Parts Choose the right examples Representative set selection Experiment design Convey the Examples Visual Comparison Ensemble Visualization
  96. 96. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 96 How do we do this? Choose the Right Examples Convey the Examples Gleicher, M. (2018). Considerations for Visualizing Comparison. IEEE Transactions on Visualization and Computer Graphics, 24(1), 413–423. InfoVis 2017.
  97. 97. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 97 A new path to model usability? Interpretable models simplify the models so they can be understood Examinable models look inside the models and hope you understand Instance-based explanations pick some decisions and try to understand them Experiment/Outcome Examination look at the right input/outputs from the black box
  98. 98. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 98 A general problem: Model Usability? How can we work with complex models? Machine learning Models of complex processes Variety of tasks Variety of users Variety of approaches Interpreting an ML model is similar to interpreting and uncertain model? Gleicher, M. (2016). A Framework for Considering Comprehensibility in Modeling. Big Data, 4(2), 75–88.
  99. 99. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 99 A Framework Complex and/or Unmeasurable Process Models Observations / Data Sample from Model Simpler/Emprical Model User Mental Model Decisions / Predictions Improved Models / Data Debugging Understanding of Variance Visualization Model Visualizaton Multi-Object Visualization
  100. 100. Michael Gleicher - Perception, Comparison, and Models for Uncertainty – Web Slides Version Talk @ EuroRV3 2019 – 4 June 2018 100 Summary Models (or lack of) are central to uncertainty but sometimes we hide them or forget them – for better/worse Communicating models may be a core problem at least it provides a way to think about things Perception and design can help communicate models but we haven’t been thinking about that enough Thinking this way may be useful for many problems but I haven’t figured it out yet Acknowledgments This work is funded in part by NSF 1162037 and DARPA FA8750-17-2-0107 Michael Gleicher Department of Computer Sciences University of Wisconsin - Madison Slides from an invited talk at EuroRV3, 4 June 2018 Not all slides were shown Photographs of other peoples’ slides used with permission

×