Developing Qualitative Metrics for Visual Analytic Environments Jean Scholtz Pacific Northwest National Laboratory BELIV 2...
Why Are Qualitative Metrics Needed? <ul><li>Quantitative metrics </li></ul><ul><ul><li>Time to accomplish a task </li></ul...
Qualitative Assessment and the VAST Challenge <ul><li>In the VAST Challenge we are able to produce quantitative metrics ea...
Evaluating the Reviews <ul><li>Our study asked the following questions: </li></ul><ul><ul><li>What materials should be pro...
What materials should be provided to evaluators to ensure that they can adequately assess the visual analytic systems?  <u...
What aspects of visual analytic systems are important to reviewers? <ul><li>We analyzed the reviews to see what comments v...
Comments on  Analytic Process <ul><li>Highly manual processes </li></ul><ul><li>Repetitive steps in process </li></ul><ul>...
Comments on Visualizations <ul><li>Complexity of visualizations </li></ul><ul><li>Misleading color coding, inconsistent us...
Comments on Interactions <ul><li>Too much scrolling </li></ul><ul><li>Interactions embedded in menus </li></ul><ul><li>Too...
Conclusions <ul><li>Reviewers were asked to comment on the categories of process, visualization and interaction </li></ul>...
Is there an advantage to selecting evaluators from different domains of expertise? <ul><li>Are there differences between w...
Why Visualization Researchers Gave Lower Ratings <ul><li>Comments </li></ul><ul><ul><li>The tool does not seem flexible en...
Overall Conclusions <ul><li>Reviewers have the appropriate material to assess the submission, assuming the material is cle...
Upcoming SlideShare
Loading in …5
×

Developing Qualitative Metrics for Visual Analytic Environments

530 views
476 views

Published on

Presenter: Jean Scholtz
BELIV 2010 Workshop Presentation
http://www.beliv.org/beliv2010/

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
530
On SlideShare
0
From Embeds
0
Number of Embeds
27
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Developing Qualitative Metrics for Visual Analytic Environments

  1. 1. Developing Qualitative Metrics for Visual Analytic Environments Jean Scholtz Pacific Northwest National Laboratory BELIV 2010
  2. 2. Why Are Qualitative Metrics Needed? <ul><li>Quantitative metrics </li></ul><ul><ul><li>Time to accomplish a task </li></ul></ul><ul><ul><li>Accuracy with which a task is done (need ground truth) </li></ul></ul><ul><ul><li>Percentage of tasks that the user is able to complete </li></ul></ul><ul><li>Qualitative information </li></ul><ul><ul><li>Provides more knowledge about what in the software actually helps the user </li></ul></ul><ul><ul><li>and how it helps the user </li></ul></ul><ul><li>Problems with obtaining qualitative information </li></ul><ul><ul><li>It is subjective </li></ul></ul><ul><ul><li>And varies by individual so generalizations are often difficult </li></ul></ul>
  3. 3. Qualitative Assessment and the VAST Challenge <ul><li>In the VAST Challenge we are able to produce quantitative metrics easily (as ground truth is embedded in the data set) </li></ul><ul><li>But what we really want to know is how well the tool, particularly the visualizations, help the user to arrive at the ground truth. </li></ul><ul><li>While we have always used reviewers (both analysts and visualization researchers) the reviews were conducted informally until 2009. </li></ul><ul><li>In 2009 we used a review system so we now have a body of reviews that we can analyze to see what is important to reviewers </li></ul>
  4. 4. Evaluating the Reviews <ul><li>Our study asked the following questions: </li></ul><ul><ul><li>What materials should be provided to evaluators to ensure that they can adequately assess the visual analytic systems? </li></ul></ul><ul><ul><li>What aspects of visual analytic systems are important to reviewers? </li></ul></ul><ul><ul><li>Is there an advantage to selecting evaluators from different domains of expertise (visualization researchers and professional analysts)? </li></ul></ul><ul><li>We analyzed the following information to answer these questions: </li></ul><ul><ul><li>Reviews (2-3 per entry) of 42 entries </li></ul></ul><ul><ul><li>Each reviewer provided a clarity rating, ratings for the usefulness, efficiency and intuitiveness of the visualizations, the analytic process and the interactions. Reviewers were also asked to rate the novelty of the submission. </li></ul></ul>
  5. 5. What materials should be provided to evaluators to ensure that they can adequately assess the visual analytic systems? <ul><li>Because of the number of teams, the research nature of the tools and the number of reviewers, it is not possible for reviewers to actually use the system. They rely on: </li></ul><ul><ul><li>A textual description, including screen shots </li></ul></ul><ul><ul><li>A video showing how a certain answer was achieved (the process) </li></ul></ul><ul><ul><li>The accuracy of the team’s answers </li></ul></ul><ul><li>And we found….. </li></ul><ul><ul><li>The materials are sufficient </li></ul></ul><ul><ul><li>The clarity of the submission definitely affects the score </li></ul></ul><ul><ul><li>Accuracy metrics impact reviewers’ scores </li></ul></ul><ul><li>Conclusions </li></ul><ul><ul><li>Emphasis to participants that they need to make sure their descriptions (text and video) are understandable </li></ul></ul><ul><ul><li>Reconsider decision to show reviewers the accuracy scores </li></ul></ul>
  6. 6. What aspects of visual analytic systems are important to reviewers? <ul><li>We analyzed the reviews to see what comments viewers made </li></ul><ul><li>We classified these comments into three categories: </li></ul><ul><ul><li>Analytic process </li></ul></ul><ul><ul><li>Visualizations </li></ul></ul><ul><ul><li>Interactions </li></ul></ul><ul><li>Notes </li></ul><ul><ul><li>There is obviously overlap so there may be disagreements about which category a comment belongs in </li></ul></ul><ul><ul><li>Most comments are stated negatively – in order to provide them in the positive we need more generalization or have to describe the actual situation (visualization, interaction or process) </li></ul></ul>
  7. 7. Comments on Analytic Process <ul><li>Highly manual processes </li></ul><ul><li>Repetitive steps in process </li></ul><ul><li>Large amount of data that analysts have to visually inspect </li></ul><ul><li>Automation that might cause an analyst not to see an important piece of data </li></ul><ul><li>Need for analyst to remember previously seen information </li></ul><ul><li>Too many steps </li></ul><ul><li>For automatic identification of patterns and/or behaviors, users need an explanation of what the software is programmed to identify </li></ul><ul><li>Analysts need to document their rationale for assumptions in their reports </li></ul><ul><li>Document the selection of a particular visualization if several are available </li></ul><ul><li>Show filters and transformations applied </li></ul><ul><li>Participants need to explain how the visualizations helped the analysis process </li></ul>
  8. 8. Comments on Visualizations <ul><li>Complexity of visualizations </li></ul><ul><li>Misleading color coding, inconsistent use of color </li></ul><ul><li>Lack of labels; non intuitive labels </li></ul><ul><li>Non intuitive symbols </li></ul><ul><li>Using tooltips instead of labels causes analysts to mouse over too many items </li></ul><ul><li>No coordination or linking between visualizations </li></ul><ul><li>The use of thickness of lines to represent strength of association is difficult to differentiate </li></ul><ul><li>Difficult to compare visualizations if can only view them serially </li></ul><ul><li>Is the visualization useful for analysis or is it a reporting tool? </li></ul><ul><li>Use of different scales in visualizations is confusing </li></ul><ul><li>Need to relate anomalies seen in visualizations to analysis </li></ul>
  9. 9. Comments on Interactions <ul><li>Too much scrolling </li></ul><ul><li>Interactions embedded in menus </li></ul><ul><li>Too many levels of menus and options to check </li></ul><ul><li>Need to be able to filter complex visualizations </li></ul><ul><li>Need to be able to drill down to actual data </li></ul>
  10. 10. Conclusions <ul><li>Reviewers were asked to comment on the categories of process, visualization and interaction </li></ul><ul><ul><li>They provided many comments but the comments were not always in the appropriate category </li></ul></ul><ul><li>Reviewers were asked to comment on efficiency, usefulness and intuitiveness </li></ul><ul><ul><li>They did mention many issues impacting each of these qualities but we did not use the individual ratings as we had expected </li></ul></ul><ul><li>For VAST 2010 Challenge we are: </li></ul><ul><ul><li>Providing the guidelines for teams (from this study plus another analyst study) </li></ul></ul><ul><ul><li>Providing definitions of process, visualization and interaction </li></ul></ul><ul><ul><li>Looking forward to identifying more guidelines (pertaining to new types of data) </li></ul></ul>
  11. 11. Is there an advantage to selecting evaluators from different domains of expertise? <ul><li>Are there differences between what visualization researchers say and what analysts say? </li></ul><ul><li>We looked at entries where there were large differences in scores between the analyst and the visualization researchers </li></ul><ul><li>Eight entries (1 analyst review, 2 researcher reviews) </li></ul><ul><ul><li>Analyst ratings were lower in 2 instances </li></ul></ul><ul><ul><li>Visualization rating was lower in 5 instances </li></ul></ul><ul><ul><li>Analyst and one visualization researcher were lower than the other visualization researcher </li></ul></ul>
  12. 12. Why Visualization Researchers Gave Lower Ratings <ul><li>Comments </li></ul><ul><ul><li>The tool does not seem flexible enough to investigate other scenarios </li></ul></ul><ul><ul><li>Should a way to quickly browse through video snippets be considered a visualization? </li></ul></ul><ul><ul><li>One visualization is provided. More are needed to look at other data. </li></ul></ul><ul><ul><li>The analytic process was not well described </li></ul></ul><ul><ul><li>Suspicious events were highlighted in the visualization but it was unclear how they were found </li></ul></ul><ul><ul><li>The visualization is distracting and not useful for analysis </li></ul></ul><ul><ul><li>The analytic process is described in terms of using this tool to detect certain kinds of events without justifying if this type of event is related to the mini challenge question </li></ul></ul><ul><ul><li>The visualization was too compressed. It was difficult to see groupings. </li></ul></ul><ul><ul><li>Tool required an iterative process and it was difficult to remember what had been done. </li></ul></ul><ul><li>Conclusions </li></ul><ul><ul><li>Visualization researchers are gaining a good understanding of what users need </li></ul></ul>
  13. 13. Overall Conclusions <ul><li>Reviewers have the appropriate material to assess the submission, assuming the material is clear and understandable </li></ul><ul><li>Analysts and Visualization researchers provide excellent comments on aspects of the system (analytic process, visualizations and interactions) although not necessarily classified correctly </li></ul><ul><li>There is little or no difference between comments provided by analysts and those provided by visualization researchers </li></ul>

×