This lecture forms part of an Information Visualisation course given at the Vrije Universiteit Brussel.

1. 2 December 2005
Information Visualisation
Analysis and Validation
Prof. Beat Signer
Department of Computer Science
Vrije Universiteit Brussel
beatsigner.com

2. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2
March 4, 2021
Four Nested Levels of Vis Design

3. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3
March 4, 2021
Validation
▪ Huge vis design space and most designs are ineffective
▪ Validate choices right from the beginning of the design
process
▪ top-down design (problem-driven work)
- start at top situation domain level
- major challenge at data/task abstraction; mainly use existing idioms
▪ bottom up design (technique-driven work)
- invention of new idioms or algorithms
▪ Validate all four levels of the design
▪ domain validation
▪ abstraction validation (what and why)
▪ idiom validation (how)
▪ algorithm validation

4. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4
March 4, 2021
Four Nested Levels of Vis Design
▪ Output from upstream level is input to downstream level
▪ errors at upstream levels cascade to downstream levels
▪ highly iterative design process

5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5
March 4, 2021
Domain Situation
▪ A domain situation is defined by
▪ target users
▪ domain of interest of target users
▪ data of target users
▪ questions (tasks) of target users
▪ Outcome of design process
▪ understanding of user needs (user-centred design)
- e.g. via observations or interviews
▪ Challenges and risks
▪ users can often not clearly specify their analysis needs
▪ designers make assumptions (rather than engaging with users)

6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6
March 4, 2021
Data & Task Abstraction
▪ Abstract from answers to domain-specific questions
at upstream to a generic representation
▪ questions from different domain situations can map to same
abstract vis tasks
- e.g. browsing, comparing or summarising
▪ Design abstract data
▪ data from upstream is often transformed to something different
▪ determine which data type supports a visual representation that
addresses a user's problem

7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7
March 4, 2021
Visual Encoding & Interaction Idiom
▪ Specific way (idiom) to create and manipulate
the visual representation of abstract data
▪ visual encoding idiom
- create a "picture" out of the data (what do users see?)
▪ interaction idiom
- how do users change what they see?
▪ Design space of the combination of visual encoding and
interaction idioms is very large
▪ data and task abstractions help to reduce the number of potential
visual encoding and interaction idioms
▪ decision about good or bad matches based on human abilities
(visual perception and memory)

8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8
March 4, 2021
Word Tree Example

9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9
March 4, 2021
Algorithm
▪ Implementation of visual encoding and interaction idioms
▪ can design different algorithms to realise the same idiom
▪ Various factors might impact the choice of a specific
algorithm
▪ computational complexity (performance)
▪ memory usage
▪ level of match with visual encoding idiom
▪ Separate algorithm design (computational issues) from
idiom design (human perception issues)

10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10
March 4, 2021
Threats to Validity
▪ Each design level has their own threats to validity
▪ wrong problem, wrong abstraction, wrong idiom or wrong algorithm

11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11
March 4, 2021
Validation Approaches

12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12
March 4, 2021
Validation Approaches …
▪ Can perform an immediate or downstream validation
▪ downstream dependencies add to the difficulty of validation
- e.g. poor algorithm design may cause doubt when testing interaction technique
▪ use of mockups for early downstream evaluation
▪ Mismatches
▪ mismatch between the level at which the benefit is claimed and
the chosen validation methodology
- e.g. benefit of new visual encoding idiom cannot be validated by measuring the
performance of the algorithm used downstream
▪ carefully select the subset of validation methods matching
the levels of design where contributions are claimed

13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13
March 4, 2021
Domain Validation
▪ A field study can help to validate that we are going
to address real user needs
▪ observe people in real-world settings
▪ semi-structured interviews (e.g. contextual inquiry)
▪ Downstream validation can for example investigate a
solution's adoption rate by the target audience
▪ see what target users do of their own accord

14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14
March 4, 2021
Abstraction Validation
▪ Identified task abstraction and data abstraction might
not solve the target audience's problems
▪ Downstream validation includes testing the solution with
members of the target audience
▪ anecdotal (qualitative) feedback whether the tool is useful
▪ field study to observe and document how the target audience
uses the tool in their real-world workflow
- observe changes in behaviour rather than documenting existing work practices

15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15
March 4, 2021
Idiom Validation
▪ Justify the design of the idiom with respect to known
perceptual and cognitive principles
▪ Heuristic evaluation or expert reviews may be used to
ensure that no known guidelines are violated
▪ Downstream validation
▪ controlled experiments in a lab setting (lab study)
- controlled experiments for testing the performance of specific idioms
- measure time and errors for given tasks
▪ presentation and qualitative discussion of results
- show images or videos of the solution to the target audience
▪ quantitative measurement of resulting visualisations (quality metrics)
such as the number of edge crossings for node-link graph
▪ usability studies

16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16
March 4, 2021
Algorithm Validation
▪ Analyse computational complexity of algorithms
▪ number of items in the dataset, number of pixels, …
▪ Downstream validation
▪ execution time
▪ memory consumption
▪ scalability
▪ Correctness of algorithm
▪ does implementation meet the idiom specification
▪ Standard benchmarks might help to compare algorithms

17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17
March 4, 2021
MatrixExplorer

18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18
March 4, 2021
MatrixExplorer Validation Methods

19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19
March 4, 2021
Genealogical Graphs

20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20
March 4, 2021
Genealogical Graphs Validation Methods

21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21
March 4, 2021
Flow Maps
Migration from California Top ten states that sent migrants to California (green)
and to New York (blue)

22. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22
March 4, 2021
Flow Maps Validation Methods

23. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23
March 4, 2021
LiveRAC

24. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24
March 4, 2021
LiveRAC Validation Methods

25. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25
March 4, 2021
LinLog Validation Methods

26. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26
March 4, 2021
Sizing the Horizon Validation Methods

27. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27
March 4, 2021
Exercise 4
▪ Analysis and Validation

28. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28
March 4, 2021
Further Reading
▪ This lecture is mainly based on the
book Visualization Analysis & Design
▪ chapter 4
- Analysis: Four Levels for Validation

29. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29
March 4, 2021
References
▪ Visualization Analysis & Design, Tamara
Munzner, Taylor & Francis Inc, (Har/Psc edition),
May, November 2014,
ISBN-13: 978-1466508910

30. 2 December 2005
Next Lecture
Data Presentation