The document summarizes an evaluation of a visualization application using Shneiderman's visualization mantra and additional heuristics. It was difficult for users to make inferences from the dense data. The evaluation tested the application's ability to provide overview, zooming and filtering, details on demand, and linking of selections. Suggestions included providing more context, statistics, and allowing saving of views and selections to link between multiple coordinated views. Limiting shape and texture options and improving color differentiation was also suggested.

1. Visualization Evaluation Utilizing the
Shneiderman Mantra
Matthew C. Doyle
State University of New York at Oswego
State University of New York at Oswego – 7060 New York 104 – Oswego, New York 1
STATE UNIVERSITY OF NEW YORK AT OSWEGO | HUMAN – COMPUTER INTERACTION

2. INTRODUCTION
The visualization walkthrough highlighted possible deficiencies in the
applications capacity to deal with the issue of information density. While the
walkthrough of the interface confirmed that users can access the information
associated with the given data points, it was still difficult to make inferences
towards possible relationships and trends held within the data. As a result, a
heuristic evaluation was conducted to focus strictly on the challenges facing users
that seek to visualize new relationships, patterns and correlations in large synoptic
sky surveys.
EVALUATION MECHANISMS
In an effort to evaluate information visualizations, one must take into
account both the usability issues of the application as well as the expressiveness and
quality of the visual representation (Freitas et al, 2002). Expressiveness and quality
can be determined by how effective the visualization is in giving expert users the
ability to gain insight about the data they are investigating. This ability comes from
giving users control over the density of both the dataset and the visualization. Drill-
down methodology can be implemented to effectively give users the opportunity to
determine the context in which reduction or refinement should occur. The
following evaluation mechanisms were chosen to gain insight into the applications
efficacy in allowing users to gain insight from the visualizations they produced.
APPLYING SHNEIDERMANS MANTRA
To establish a practical approach for evaluating information visualizations,
Ben Shneiderman established the Visual Information-Seeking Mantra (1996) to
describe the functionality thata visualization technique should provide
(Shneiderman, 1996). The basic principles of this methodology consist of providing
the user with an overview of the data, allowing the user to selectively zoom and
filter relevant information, bring up details about the dataset based on a specified
data points, and highlight a specific subset data points and view them separately
through different visualizations (Craft & Cairns, 2005). Together, these ideals are
designed to allow the users to drill into a large dataset and retrieve bits of
interesting information to be compared on a smaller scale.
OVERVIEW
Description: In the overview, the user should be able to identify interesting
patterns and focus on one multiple more closely. Significant features can be isolated
and selected for further examination, aiding the user in filtering extraneous
information so that they can complete their task more efficiently by excluding
unimportant aspects of the representation (Stephens, 2003).
Analysis: The iViz application currently allows the user to browse through a large
dataset using a 3D panoramic viewing perspective. Users can also filter extraneous
information by zooming in on a particular subset of data, unchecking shape and
State University of New York at Oswego – 7060 New York 104 – Oswego, New York 2
STATE UNIVERSITY OF NEW YORK AT OSWEGO | HUMAN – COMPUTER INTERACTION

3. texture parameters, and using the each function at the bottom of the screen.
Suggestions: Users may want to know some overlying statistics about the dataset
as they browse so they have a better idea of the data and can detect outliers quicker.
ZOOM & FILTER
Description: Zooming and filtering are also crucial techniques that can be used to
overcome information density. Zooming has two functions; to display the data
objects larger and to present additional details about the data as it zooms in.
Filtering allows the user to hide or reveal data of interest so the information can be
simplified to aid cognition. Dynamic filtering allows users to quickly see how the
changed variable affects the data visualization with search filters. Dynamic queries
allow the user to adjust the parameters of a database query in order to return
results by keyword, category, range, date, etc. (Craft & Cairns, 2005).
Analysis: Zooming and filtering techniques are both utilized in the iViz application.
Users can zoom throughout the dataset in 3D and filter information by checking or
unchecking shape or texture features. There is also a search function that allows the
users to filter out certain data points that don't fit a range specified by the user.
Suggestions: The zooming affordance works well but it would be helpful to allow
the users to hit a button where they can return to their original position when
zooming or save their current perspective. Zooming could also bring in additional
information about the data points, such as descriptive statistics. The system could
make better use of dynamic filtering by allowing the users to filter by a range of
values, keywords, categories, or dates.
DETAILS ON DEMAND
Description: Details on demand allow users interactively select parts of data to be
visualized more detailed while providing an overview of the whole informational
concept. This technique also provides supplementary information on a point-by-
point basis without requiring a change of view. This is useful for relating the
detailed information to the rest of the data set or for quickly solving particular tasks,
such as identifying a specific data element amongst many, or relating attributes of
two or more data points.
Analysis: The iViz application provides these details when a user selects a specific
data point. From here, a window emerges with all of the variable information
associated with the data point. This allows users to uncover new information
without changing the representational context in which the data is arranged.
Suggestions: The application would benefit from providing descriptive statistics
and context towards the variables that may hold outliers. The details on demand
feature should allow users to begin to discover data points they might want to
highlight and visualize further.
State University of New York at Oswego – 7060 New York 104 – Oswego, New York 3
STATE UNIVERSITY OF NEW YORK AT OSWEGO | HUMAN – COMPUTER INTERACTION

4. LINKING & BRUSHING
Description: Connecting multiple visualizations through interactive linking and
brushing provides more information than considering the larger visualization
independently (Keim, 2002). As a result, the idea of linking and brushing is to
combine different visualization methods to overcome the shortcomings of single
techniques.
Analysis: The application currently does not allow the user to highlight specific
data points and visualize them separately from the entire dataset.
Suggestions: Many users insisted that they would like to see a feature that allows
them to select a group of data points and visualize them in different ways. It would
be helpful to allow users to save subsets of data points while browsing through the
dataset and apply them to linked visualizations to aid in the discovery of
dependencies and correlations.
ADDITIONAL EVALUATION CRITERIA
In “Evaluating usability of information visualization techniques” Frietas et al
(2002) proposed three additional evaluation parameters to measure the
effectiveness of a visualization space. These metrics include completeness, spatial
organization, codification of information, and state transition. Alongside the
Shneiderman Mantra, these heuristics serve as a suitable foundation for
visualization evaluation.
COMPLETENESS
Description: The concept of representing all the semantic contents of the data to be
displayed. This is affected by the geometric or visual constraints (size of the display,
maximum number of data elements, etc.) imposed by the visual representation as
well as by its cognitive complexity, which in turn can be measured by data density,
data dimension and by the relevance of the displayed information (Freitas et al,
2002).
Analysis: The current iteration of this system suffers from information overload in
that it overwhelms the user with a lot of information with limited viewing
perspectives and no affordances for selecting specific data points and highlighting
them to view them from multiple viewpoints and different visual representations.
Suggestions: Users should be able to extract important findings and save their
work to avoid unnecessary redundancy. Allow extraction of sub-collections and of
query parameters.
SPATIAL ORGANIZATION
State University of New York at Oswego – 7060 New York 104 – Oswego, New York 4
STATE UNIVERSITY OF NEW YORK AT OSWEGO | HUMAN – COMPUTER INTERACTION

5. Description: Related to the overall layout of a visual representation, which
comprises analyzing how easy is to locatean information element on the display and
to be aware of theoverall distribution of information elements in the representation.
The spatial orientation, which contributes for the user being aware of the
distribution of information elements, is dependent on the presentation of context
while displaying a specific element in detail (Freitas et al, 2002).
Analysis: Some data points overlap each other, making it difficult to differentiate
some points from others. The shape and texture elements are sometimes troubling
for users because they cannot differentiate between the two classifiers. Utilization
of poor color also makes some data points hard to see.
Suggestions: Users would benefit from the simplification of shape and texture
parameters that are available for variable assignment. Many users had trouble
distinguishing between similar shapes and textures. The application should use
three to four different shapes and no more than two textures.
CODIFICATION OF INFORMATION
Description: The use of additional symbols or realistic characteristics can be used
either for building alternative representations (like groups of elements in clustered
representations) or to aid in the perception of information elements (Freitas et al,
2002).
Analysis: The application currently utilizes location (XYZ), color (RGB), shape,
texture, and opacity to codify variables into visual representations. Users have
expressed difficulty distinguishing between elements due to poor use of color and
texture.
Suggestions: Although variables can be given many different visual attributes, the
data can still overwhelm the user. At times users struggled to differentiate between
shapes and textures, which led them to select the wrong class when attempting to
retrieve information from a specific star. iViz should primarily rely on color to
group clusters of information together and visualize them in small multiples, using
fewer shapes and textures as classifiers.
STATE TRANSITION
Description: The result of rebuilding the visual representation after a user action.
The time spent by the technique to do that and the changes in spatial organization of
the resulting image are important factors that can affect the perception of
State University of New York at Oswego – 7060 New York 104 – Oswego, New York 5
STATE UNIVERSITY OF NEW YORK AT OSWEGO | HUMAN – COMPUTER INTERACTION

6. information (Freitas et al, 2002).
Analysis: Currently, the application allows users to change state by zooming in and
out of a large 3D data set represented as a bubble chart, but there is no mechanism
for users to revert back to their original position or to a specific view. As a result, it
can be difficult to recreate exact visualizations.
Suggestions: Users would benefit from the ability to revert to a familiar spot in case
they get lost or to go to a specified position in the 3D visualization to recreate exact
perspectives. The ability to take screenshots of a perspective in the 3D
representation or of a specific visualization could also help users share information
with collaborators or save it for future investigation.
OVERVIEW OF SUGGESTIONS
Overview is given but participants of the walkthrough oftentimes got
confused while exploring in 3D. Many users wanted the opportunity undo or redo
function to revert back to a previous view of the data. Zooming functionality
allowed users to identify a specific data point that adheres to specific constraints.
Further, the capability to generate details on demand from clicking on a specific data
point was efficiently integrated into the application. However, users were not able
to highlight interesting data points and link them together to create lists.
It would be helpful for expert users to gain insight towards possible
relationships within the data by allowing them to explore the dataset in an
immersive 3D representation while being able to receive details on demand on
specific data points and save ones that are of interest. Users could then link these
data points together, and visualize the across multiple small visualizations. This
would allow the user to drill down into the data and break it down into small linked
fragments of information.
VISUALIZATION
Users suggested adding context to what certain variables represented in the
data set, as well as the ability to preview the visualization before choosing to map a
certain variable to a given attribute. Lastly, users also requested the ability to
compare two different sets of visualizations at once. This ability could be
implemented by a mechanism that provides small multiples of information for the
user.
Small Multiples
Following the proposed interface functionality of filtering, brushing, and
linking, small multiples would be helpful to aid users in reducing information
density by using the linked data points to generate quick comparisons amongst each
other.
State University of New York at Oswego – 7060 New York 104 – Oswego, New York 6
STATE UNIVERSITY OF NEW YORK AT OSWEGO | HUMAN – COMPUTER INTERACTION

7. Variety of Visualizations
From this, it would be helpful to offer users the opportunity to visualize
information in representations that can provide information at a glance as well as
advanced visualizations that can provide additional insight into relationships
amongst variables in the dataset.
Classifiers
Users had a hard time determining the difference between textures.
No noisy fill patters or line styles. Others did not find the shapes helpful as
identifiers because they were hard to differentiate between. Users indicated that
they would like to be able view corresponding colors to the different star types as
classified by shape. Others would like to select a group of data points and
manipulate them separately.
Limit the amount of options for users to encode texture and shape into classifiable
variables.
Color
Users did not find the shapes helpful as identifiers because they were hard to
differentiate between. Further, users also indicated that the colors of some data
points were too similar and found it hard to differentiate between them at times. No
saturated or bright colors
State University of New York at Oswego – 7060 New York 104 – Oswego, New York 7
STATE UNIVERSITY OF NEW YORK AT OSWEGO | HUMAN – COMPUTER INTERACTION