Pragmatic Issues in the Evaluation of Interactive Visualization Systems April 10, 2010 Youn-ah Kang, Carsten Görg, John St...
<ul><li>A comparative lab study with  an investigative analysis scenario </li></ul>VAST 2009
Paper Desktop Entity Jigsaw
Pragmatic Challenges <ul><li>1. Student as Subject </li></ul><ul><ul><li>Graduate students vs. domain experts </li></ul></...
Pragmatic Challenges <ul><li>3. Statistical Power vs. Analysis Depth </li></ul><ul><ul><li>Hard to have both large partici...
Pragmatic Challenges <ul><li>5. Familiarity with the Tool </li></ul><ul><ul><li>Good usability is critical in a utility ev...
Answers <ul><li>??? </li></ul>
Upcoming SlideShare
Loading in …5
×

Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.

406 views

Published on

Presenter: John Stasko, Youn-ah Kang, Carsten Gorg
BELIV 2010 Workshop
http://www.beliv.org/beliv2010/

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
406
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Pragmatic Challenges in the Evaluation of Interactive Visualization Systems.

  1. 1. Pragmatic Issues in the Evaluation of Interactive Visualization Systems April 10, 2010 Youn-ah Kang, Carsten Görg, John Stasko School of Interactive Computing Georgia Institute of Technology
  2. 2. <ul><li>A comparative lab study with an investigative analysis scenario </li></ul>VAST 2009
  3. 3. Paper Desktop Entity Jigsaw
  4. 4. Pragmatic Challenges <ul><li>1. Student as Subject </li></ul><ul><ul><li>Graduate students vs. domain experts </li></ul></ul><ul><ul><li>Difficulty in accessing domain experts </li></ul></ul><ul><ul><li>To what extent are the results valid? </li></ul></ul><ul><li>2. Quantitative vs. Qualitative </li></ul><ul><ul><li>Are the two really incompatible? </li></ul></ul><ul><ul><li>Using a mixed method to further validate and enrich findings </li></ul></ul>
  5. 5. Pragmatic Challenges <ul><li>3. Statistical Power vs. Analysis Depth </li></ul><ul><ul><li>Hard to have both large participant population and in-depth analysis of each session </li></ul></ul><ul><ul><li>Need to focus on either side </li></ul></ul><ul><li>4. Scalability </li></ul><ul><ul><li>Balancing between feasibility and ecological validity </li></ul></ul><ul><ul><li>Chose the document size to make the study feasible in a given time, but in reality… </li></ul></ul>
  6. 6. Pragmatic Challenges <ul><li>5. Familiarity with the Tool </li></ul><ul><ul><li>Good usability is critical in a utility evaluation </li></ul></ul><ul><ul><li>Unfamiliarity with the tool blurs potential benefits </li></ul></ul><ul><ul><li>Big learning curve but limited time </li></ul></ul><ul><li>6. Comprehensiveness of the Task </li></ul><ul><ul><li>Only targeting scenario for this study </li></ul></ul><ul><ul><li>Challenges in developing evaluation strategies and measures for less focused scenarios </li></ul></ul>
  7. 7. Answers <ul><li>??? </li></ul>

×