1) The study tested how expert users interact with animated versus non-animated maps to understand the narrative of floods impacting a railway network. 2) The results found no significant difference in efficiency between map types, but animated maps influenced user confidence and hesitation. Information retrieval tasks were easier than analyzing causality. 3) Two questions had ambiguous wording, influencing the results. Additionally, question complexity likely had a greater impact than map type. 4) Future work could involve controlling task types by using standardized sets from other domains to improve reproducibility and comparison across studies.

1. User Testing of Dynamic Geovisualizations:
Lessons Learned and Possible Improvements
for Cartographic Experiments
Cécile Saint-Marc,1 Marlène Villanova-Oliver,1 Paule-Annick Davoine,1 Cicely Pams-Capoccioni,2 Dorine Chenier2
1. Univ. Grenoble Alpes - CNRS - LIG
2. SNCF Ingenierie & Projets

2. Context
• Test ways to visualize domino effects occuring during risk
events
– Case study : floods impacting the railway network
09/08/2017 2
Heavy rain
Water rise
Too much
vegetation Earthwork
submersion
Erosion
Railway destruction
Works

3. A recurring research question
Are animated maps more efficient than non-animated maps
to display temporal phenomena ?
 An intractable issue
(inconsistent experimental results)
Because :
• It depends on the context of use (audience, task, aim) (Lowe 1999)
• It depends on the experimental procedure (wording of questions,
differences between tested maps) (Tversky et al. 2002)
• Animations inherently trigger different cognitive processes in
comparison with static views (Fabrikant et al. 2008)
09/08/2017 3

4. A given use case
• One audience :
– Expert users
• One aim :
– Understand the narrative of a flood and the damages it caused to the
railway network
• Comparable maps :
– One geovisualization interface, all the events displaid VS. animated
appearance of events
• Given tasks :
– Information retrieval
– Reading of causality relationships between events
• Careful reasoning about the wording of questions
09/08/2017 4
Still some limitations
in the results !

5. Questions ?
• What went wrong ?
• Is it avoidable ?
09/08/2017 5

6. Plan
1. Experimental procedure
2. Results
3. Limitations
4. Outlooks
09/08/2017 6

7. Material
09/08/2017 7
1. Experimental procedure

8. Material
09/08/2017 8
1. Experimental procedure

9. Hypothesis
The animated map is more efficient than the interactive map
to perceive causality relationships between events.
• Efficient :
– Reach the objective
– By spending the less ressources (time)
09/08/2017 9

10. Procedure
Independent variables :
• Map type
– Animated
– Non-animated
• Question aim
– Information retrieval
– Causality reading
• Information dimension adressed by the question
– Attribute
– Space
– Time
– Space & Time
09/08/2017 10

11. Procedure
09/08/2017 11
• Introduction : Explanations, Demographic questionnaire
• Tasks
Set 1 : Information retrieval
Set 2 : Causality reading
Tutorial
1 Training question
2 Warm-up questions
3 Task-typed questions
Tutorial
1 Training question
2 Warm-up questions
3 Task-typed questions
Opinion questionnaire
Map type A (or B)
Map type B (or A)

12. Tasks
09/08/2017 12
Task type Code Question Dimension
Warm-up TF1 Which date is represented by orange color in the
legend?
Semiology
Warm-up TF4 What is the symbol representing a damage on an
earthwork?
Semiology
Warm-up TF5 What is the date of the first work event? Time
Warm-up TF3 At which kilometric point do the event of November
21st occurred?
Space x Time
Retrieval T2 Was level crossing 264 overflowed by the river? Space
Retrieval T1 How many events of type ‘work’ happened on
November 14th?
Time
Retrieval T3 When were undermined the riprap protecting the
viaduct of Orbieu?
Space x Time
Causality T5 What was the consequence of the event that occurred
at kilometric point 391+760?
Space
Causality T8 When have the posts of new overhead line masts
finished?
Time
Causality T7 How many direct causes and consequences are linked
to the event of November 16th located on the viaduct
of Orbieu?
Space x Time

13. Procedure
Dependent variables :
• Response accuracy
• Response time
• Duration before the first click
• Deviation to the optimal number of clicks
09/08/2017 13

14. Plan
1. Experimental procedure
2. Results
3. Limitations
4. Outlooks
09/08/2017 14

15. Main results
• No significant difference of efficiency between the interactive map and
the animated map (hypothesis rejected)
– But differences in users’ confidence in their answers and hesitation before interacting
• Information retrieval tasks are easier to answer than causality reading
tasks (86% vs. 62% of correct answers / 79’’ vs. 104’’)
• Tasks questioning both space and time are significantly more difficult to
answer
• Complementary variables monitored are interesting for the analysis
09/08/2017 15

16. Plan
1. Experimental procedure
2. Results
3. Limitations
4. Outlooks
09/08/2017 16

17. Two imprecise questions
• Questions T2 and T8 were thematically ambiguous, from an
expert point of vue
• Not detected during the pre-test
• Influence on the results
– Longer response time
– Random answers
 T2 and T8 removed from the results analysis
 Warm-up tasks included in the analysis
09/08/2017 17

18. Unexpected factors
09/08/2017 18

19. Unexpected factors
• Impact of the « cognitive complexity » of questions
(Anderson and Krathwohl 2001)
09/08/2017 19
6 levels :
• Remember
• Understand
• Apply
• Analyze
• Evaluate
• Create

20. So…
• Quantitative results seems to be more related with the type
of questions than with the type of map ! (primary dependent
variable)
Need to control the type of question for the inner consistency
of studies…
… And in order to compare the results of various studies
09/08/2017 20

21. Plan
1. Experimental procedure
2. Results
3. Limitations
4. Outlooks
09/08/2017 21

22. Need for controlling the types of tasks
• Already begun by Golebiowska et al. (2016)
09/08/2017 22

23. But a N-complex problem
• Typologies of experimental tasks in cartography :
– Wehrend and Lewis (1990) : locate, distinguish, analyze the data
distribution, and more…
– Amar et al. (2005) : retrieve a value, filter, characterize data distribution,
correlate, find anomalies, and more…
– Gotz and Zhou (2009) : data exploration (ask, filter), visual exploration
(browse, zoom, categorize), visual deduction (annotate, mark), and more…
– Edsall (2003) and Tobón (2005) : question one or many objects X one or
many variables
– Bertin, in Bianchin (2002) : elementary, intermediate or global level of
question
– Information dimension (tested here) : attribute, space, time and their
combinations
– Anderson and Krathwohl (2001) (tested here) : cognitive complexity of
questions
– and others…
09/08/2017 23

24. A possible solution
• Other research fields (ex: information retrieval – see Wu et al. 2012;
Moffat et al. 2013) share standardized sets of tasks to improve
reproducibility of results
– In several thematic fields such as health, leisure, sport…
• Advantages :
– Solve the N-complex problem of designing tasks sets only once, twice
or not a lot more, for each thematic field
– Improve the robustness of task sets by increasing the number of users
and contributors
– Improve comparison between experimental results
09/08/2017 24

25. A first look for cartographic experiments
09/08/2017 25
Share sets of tasks in most
common thematic fields
Share data sets in most
common thematic fields too
Test new geovisualization proposals
on these standardized sets
Compatible with a second test on individual
and specific use cases
Comparison between
studies !

26. Thank you for listening !
Some questions ?
Contact : cecile.saint-marc@altametris.com
09/08/2017 26