Learning-Based Evaluation of Visual Analytic Systems.

794 views

Published on

Presenter: Remco Chang
BELIV 2010 Workshop
http://www.beliv.org/beliv2010/

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
794
On SlideShare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Learning-Based Evaluation of Visual Analytic Systems.

  1. 1. Learning-Based Evaluation of Visual Analytics Systems Remco Chang, Caroline Ziemkiewicz, Roman Pyzh, Joseph Kielman*, William Ribarsky UNC Charlotte *Department of Homeland Charlotte Visualization Center Security
  2. 2. Why Another Evaluation Method? • Based on a discussion with Joe Kielman (DHS) – Why is it difficult for agencies like the DHS to adopt and use visual analytics systems? • Most existing metrics are not indicative of success of adoption – Task completion time – Errors – Subjective preferences – Etc.
  3. 3. Current Methods • Methods for evaluating visual analytics systems have been proposed. Each has its unique perspective and goal. For example: – Insight-based Evaluation (North et al.) – Productivity-based Evaluation (Scholtz) – MILC -- Multi-dimensional in-depth long-term case studies (Schneiderman, Plaisant) – Grounded Evaluation (Isenberg et al.)
  4. 4. Our Goal for Evaluation • What Joe wants is: – Proof that the user of the visual analytics system can gain proficiency in solving a problem using the system – By using the VA system, show that a user can gradually change from being a “novice” to becoming an “expert” • In other words, Joe wants proof that by using the VA system, the user is gaining knowledge… – The goal of visualization is to gain insight and knowledge (ViSC report, 1987) (Illuminating the Path)
  5. 5. Learning-Based Evaluation • In light of this goal, we propose a “learning-based evaluation” that attempts to directly test the amount of knowledge gained by its user. • The idea is try to determine how much the user has learned after spending time using a VA system by: – Giving a user a similar but different task. – Directly testing if the user has gained proficiency in the subject matter.
  6. 6. Current Method
  7. 7. Our Proposed Method
  8. 8. Types of Learning • In designing either a new task or the questionnaire, it is important to differentiate and isolate what is being tested: – Knowledge gained about the Interface – Knowledge gained about the data – Knowledge gained about the task (domain)
  9. 9. iPCA Example • iPCA stands for “interactive Principle Component Analysis”. By using it, the user can learn about: – The interface – The dataset • relationships within the data – The task • What is principle component analysis, and • How can I use principle component analysis to solve other problems?
  10. 10. Application to the VAST Challenge • Current method: – Give participants a dataset and a problem – Ask participants to develop VA systems to solve the problem – Ask participants to describe their systems and analytical methods – Judges score each submission based on the developed systems and their applicability to the problem
  11. 11. Application to the VAST Challenge • Proposed method: – Give participants a dataset and a problem – Ask participants to develop VA system to solve the problem – Ask participants to bring their systems to VisWeek – Give participants a similar, but different dataset and problem – Ask participants to solve the new problem using their VA systems – Judges score each participant based on the effectiveness of each system in solving the new task.
  12. 12. Types of Learning • In designing either a new task or the questionnaire, it is important to differentiate and isolate what is being tested: – Knowledge gained about the Interface – Knowledge gained about the data – Knowledge gained about the task (domain)
  13. 13. Discussion/Conclusion • This learning-based method seems simple and obvious because it really is. Teachers have been doing this for ages. • The method is not unique. There are many aspects of this proposed method that are similar to existing methods. In spirit, we are all looking to address the same problem. • The difference is the perspective. If we think about the problem from the perspective of a client (e.g., Joe at DHS), what they look for in evaluation results currently are not the same as what we as researchers give them.
  14. 14. Future Work • Integrate the proposed learning-based method to: – Grounded Evaluation – Long term effects (MILC)
  15. 15. Thank you! rchang@uncc.edu http://www.viscenter.uncc.edu/~rchang
  16. 16. The Classroom Analogy • Say you’re a math teacher in middle school, and you’re trying to decide which text book to use, the blue one or the red one. You can: – Ask your friends which book is better • Analogous to an “expert-based evaluation”. Problem is that the sample size is typically small, and the results difficult to replicate. – Ask your students which book they like • Analogous to subjective preferences. The issue here is that the students can prefer the blue text book because its blue. – Test which text book is more effective by giving the students tests.

×