Comparing Benchmark Task and Insight Evaluation Methods.


Published on

Presenter: Chris North, Purvi Saraiya
BELIV 2010 Workshop

  • Be the first to comment

  • Be the first to like this

Comparing Benchmark Task and Insight Evaluation Methods.

  1. 1. Comparing Benchmark Task and Insight Evaluation Methods Purvi Saraiya, Chris North, Karen Duca Virginia Tech BELIV 2010
  2. 2. Single-Time Viz Multi-Time Viz Multi-Graph Viz 1 2 3 4 5 6 (Timepoints) Non-Smokers Smokers Non-Smokers Smokers 1 2 3 4 5 6 (Timepoints)
  3. 3. Comparison of Results Benchmark task method result Insight method result Confirm Timepoint analysis on Multi-Time Viz Refute Topology tasks on Multi-Graph Viz Less insight Extend Hypothesis tasks on Single-Time Viz Outlier node tasks on Multi-Time Viz Refute ??? Confirm Timepoint analysis on Single-Time Viz More insight Not tested Slow / inaccurate Fast / accurate
  4. 4. Other interesting stuff <ul><li>Importance of interaction to insight </li></ul><ul><li>Subjectivity Threat </li></ul><ul><li>Task method = choice of tasks ecological validity </li></ul><ul><li>Insight method = coding of insights repeatability </li></ul>
  5. 5. Question <ul><li>Which is more important to know: </li></ul><ul><li>Tasks a visualization supports </li></ul><ul><li>or </li></ul><ul><li>Tasks a visualization promotes </li></ul><ul><li>? </li></ul>
  6. 6. Studies using VAST Challenge
  7. 7. Controlled Ethnography !? <ul><li>VAST Challenges – overcoming the problems with ethnography (esp. in intell) </li></ul>(contextual) interviews observational lab study Gain rich insight into analysts’ processes, Quantify findings with repeatability, Introduce interventions generate scenario confirm findings