This thesis explores how visualizing complex health data can aid clinicians in understanding patient risk factors through sensemaking theory and visualization techniques. The study presents findings from an exploratory research project that developed and tested a visual tool designed to facilitate this process, highlighting two key findings about the multifaceted nature of risk assessment and the tool's potential to externalize implicit clinical processes. The document discusses the methodology, design of the tool, and implications for improving clinical decision-making through enhanced information visualization.

1. MAKING SENSE OF RISK BY VISUALIZING
COMPLEX HEALTH DATA
Nicholas Tenhue
MSc ICT Innovation
University College London, 2014

3. MAKING SENSE OF RISK BY VISUALIZING
COMPLEX HEALTH DATA
Nicholas Tenhue
MSc ICT Innovation & Data Vis & Design Intern, Intel Health & Life Sciences
Author and Principal Investigator
Ann Blandford
Professor of Human-Computer Interaction
Thesis Supervisor
Chiara Garattini
Anthropology & UX Research, Intel Health & Life Sciences
Industry Supervisor
This report is submitted as part requirement for the MSc Degree in ICT Innovation,
at University College London. It is substantially the result of my own work except
where explicitly indicated in the text.
The report may be freely copied and distributed provided the source is
explicitly acknowledged.
September 2014
University College London, 2014

5. 1
MAKING SENSE OF RISK BY VISUALIZING
COMPLEX HEALTH DATA
Methods for visualising patient data in a way that supports sensemaking
may help clinicians to understand risk factors at the individual patient level.
This thesis uses sensemaking theory and visualisation techniques to de-
velop a tool and test it with clinicians in the healthcare domain. This is an
exploratory study into how information visualisation techniques can help cli-
nicians make sense of risk in a patient. We present a chronological account
of the approach taken to build and assess a visual tool for sensemaking.
We present two main findings (i) making sense of risk is a multifaceted pro-
cess that entails complexity beyond just using research evidence and clini-
cal expertise (ii) we have preliminary evidence that the visual tool supports
by creating externalisations that facilitate to make the implicit processes
that they use frequently in their work, explicit.
Keywords: information visualisation, sensemaking, design

6. 2

7. 3
I owe this journey of discovery and learning to a number of people.
Academic Supervisor, Ann Blandford, who gave sound advice throughout.
Industry Supervisor, Chiara Garattini, for advising me with her deep
knowledge of anthropology and UX.
Mentors, Mario Romero and Connor Upton, who shared their expertise in
information visualisation.
My peers Misha Patel, Hanna Schneider and David Pribil for their constant
encouragement.
This thesis is written and reported, in entirety, by the Author. However,
employees at the Author’s internship company provided invaluable
assistance:
 Chiara Garattini – risk calculator research & persona creation; partici-
pants’ recruitment;
 Marisa Parker – assisting in design of Stage 2 & 3 prototypes;
 Reese Bowes – final screenshots for use in external publication.

8. 4

9. 5
TABLE OF CONTENTS
1 Introduction..........................................................................................7
2 Background .......................................................................................11
2.1 Sensemaking..............................................................................11
2.2 Visualisation for sensemaking ....................................................14
3 Methods.............................................................................................21
3.1 Participants.................................................................................21
3.2 Apparatus & Materials ................................................................22
3.3 Ethical considerations.................................................................25
4 Design, study & analysis....................................................................27
4.1 Patient personas.........................................................................27
4.2 Stage 1.......................................................................................29
4.3 Stage 2.......................................................................................33
4.4 Stage 3.......................................................................................48
5 Results ..............................................................................................61
5.1 Data complexity & the implicit.....................................................61
5.2 Clinical workflow.........................................................................64
5.3 Thinking about risk .....................................................................66
5.4 Dealing with data........................................................................68
5.5 Visualising risk............................................................................74
6 Limitations .........................................................................................83
6.1 Design failures in the tool ...........................................................83

10. 6
6.2 Limitations of the study...............................................................85
7 Conclusions.......................................................................................87
8 Bibliography.......................................................................................89
9 Appendices........................................................................................97
9.1 Appendix A – Participant Information..........................................97
9.2 Appendix B – Consent Form.....................................................100
9.3 Appendix C – Interview Plan (Stage 2) .....................................101
9.4 Appendix D – Interview Plan (Stage 3) .....................................104
9.5 Appendix E – Requirements Statement ....................................105

11. 7
1 INTRODUCTION
When dealing with risk the clinician must search for and decide what data is rele-
vant to the individual patient. This complex task involves judging the integrity of
the data, the relevance of that data to the individual patient, the particulars of the
circumstances, patient wishes, and a host of other variables. Few tools support
this process because they downplay the role of clinical expertise for judging par-
ticular circumstances, instead they rely only on empirical population studies that
may or may not apply to the individual in question and leave the rest to the clini-
cian.
Clinicians are forced to deal with an excess of data in their work (Feblowitz,
Wright, Singh, Samal, & Sittig, 2011). This issue will continue to grow as a sur-
plus of noisy, multivariate, homogeneous data is generated from a number of dis-
similar sources. Clinicians already deal with patient reported data and hospital
generated data. Also, the proliferation of wearables and mobile devices lead us to
believe that doctors are increasingly exposed to self-generated health data (My-
natt, 2011), providing new insight into patients’ lives that population studies. Most
of the existing risk algorithms and evidence-based research do not presently take
data generated from these new types of eHealth self-monitoring devices into ac-
count. In addition, recent developments in whole genome sequencing and ge-
nomic science are making personalised healthcare possible. There is potential to
integrate this wealth of data with traditional health data, to present a tailored rep-
resentation of patient. Nevertheless, the question arises of how the clinician
makes sense of all of these factors to form an overarching understanding of risk
in an individual patient.
Patient information is highly complex with data intervals ranging from minutes to
decades (Shneiderman, Plaisant, & Hesse, 2013). The best possible care is de-
livered when clinicians can, without difficulty, consolidate and make sense of this
patient information in a way that matches their mental model (Johnson-Laird,
1983). Clinicians must seek out data and organize it internally to form a unified

12. 8
understanding of the patient’s current condition (Faiola & Hillier, 2006), this in-
creases cognitive load and the time spent foraging for relevant information.
Possibly more problematic is the lack of externalisations that show relational or
context-based data (Faiola & Hillier, 2006) that would allow clinicians to recog-
nise trends and relationships between co-variables. This leads to the question
‘How can we inform, rather than overwhelm clinicians when they are faced with
these problems?’
One way to tackle this issue is through the use of digital, interactive, visual repre-
sentations of data (Card, Mackinlay, & Shneiderman, 1999). Through the effec-
tive use of visualisations it is possible to deliver what Spence (2007) calls an ‘A
Ha!’ moment, providing insight to task-specific problems that clinicians face. Yet,
information visualisation alone is not the answer. The solution must also support
clinicians’ sensemaking whilst focusing on the quality of the fit between the user
and system models.
This thesis expands on existing knowledge within information visualization and
sensemaking literature and attempts to apply it to creating a tool to support clini-
cians who deal with complex health data when assessing risk.
The aims of this thesis are twofold:
 Firstly, we aim to provide a domain specific account of the way in which clini-
cians make sense of complex health data when assessing risk in a patient.
 Secondly, by understanding clinicians’ needs and practices we aim to itera-
tively develop a visualisation tool with representations that reduce the gap
between the data and the clinicians’ mental model.
The goal of this thesis is to stimulate discussion in the human-computer interac-
tion (HCI) and information visualisation communities by contributing to the under-
standing of how visualization of complex health data can support clinicians in
making sense of a patient’s risk. To further this goal, this thesis presents findings
from an exploratory study where a tool was designed to support clinicians’ needs
during the assessment of risk.

13. 9
Chapter 2 begins with a review of relevant literature in sensemaking and infor-
mation visualisation. We discuss how visualisations can be powerful tools for
making sense of a domain, and then look at literature on how to design visualisa-
tions. In Chapter 3, the methods used in the study are presented along with the
rationale behind using them. Chapter 4 is a chronological account of the project.
The design process of the tool created for the study, study procedure, and ap-
proach to analysis of each stage are described. The HCI and information visuali-
sation methods that were used are reflected upon. Findings and discussion are
combined in Chapter 5, where we first talk about how clinicians’ understand and
think about the domain, and then give examples of where the tool supported this.
We discuss the limitations of the study and the tool in Chapter 6, along with pos-
sible areas for further investigation. Finally, we draw conclusions in Chapter 7.

14. 10

15. 11
2 BACKGROUND
This chapter provides an outline of the existing work relevant to clinicians making
sense of risk and visualising complex health data. Also, the motivations behind
using that work are given. Firstly, literature on sensemaking and existing models
are explored. Secondly, we look at how visualisation can play a role in making
sense of a domain. Finally, information visualisation design considerations and
techniques required to create the tool in this thesis are discussed.
Sensemaking is regarded as the process of information seeking and interpreta-
tion, it is about how people make sense of and understand a domain or topic, in
this case clinicians making sense of risk in a patient. Sensemaking research is
employed in a number of disciplines (e.g. decision-making, organisational re-
search), here we use it specifically to look at how the individual clinician makes
sense of risk. Klein, Moon, & Hoffman (2006) describe sensemaking in modern
research as a continuous effort to comprehend connections between individuals,
places, and events in order to anticipate their trajectories and act accordingly.
There have been a number of efforts to
formalise this process in sensemaking re-
search. In this section we will delve into
some of the leading theories in sense-
making and discuss their relevance to this
thesis as well as how looking through the
sensemaking lens might be beneficial in
understanding the problem domain.
Russell, Stefik, Pirolli, & Card (1993)
describe the sensemaking process
through what they call the Learning
Loop Complex model (Figure 1).
Figure 1 the Learning Loop Complex.
Image taken from Russell et al.
(1993).

16. 12
This model follows a pattern where information in encoded in representations to
reduce the cost of operations. This involves four sensemaking phases:
 Search for good representations – representations are created to track
regularities that are significant to the sense maker. This is the generation loop.
 Instantiate representations – significant information is identified and encoded
in a suitable representation. Encodons are created in the data coverage loop.
 Shift representations – data (residue) that does not fit the existing schema
force a change in representations by moving up through the representational
shift loop, leading to merging, division and generation of schema.
 Consume encodons – encodons are used in task-specific information
processing.
The Learning Loop Complex model shows how sense makers use a top-down
(representation instantiation) and bottom-up (representation search) process to
form a mental model of a domain.
Pirolli & Card (2005) identify sensemaking in terms of two loops; the foraging and
sensemaking loops. The foraging loop includes three processes; exploring, en-
riching and exploiting. Exploring is about searching a space to gain new infor-
mation. In the setting of healthcare, this could mean retrieving doctor’s notes and
health record data. Through enriching, a clinician might order new investigations
or drill deeper into information to come up with a higher precision account of a
patient. Exploiting items in a set could mean going through patient information
and making inferences and detecting patterns. The sensemaking loop involves a
recurring process in which a mental model that matches the evidence is created.
However, these models do not take into account interaction effects that can occur
in sensemaking. Weick (1996) notes that schemas do not shift easily as residue
goes unnoticed by the sense maker. People to people interactions can cause
representational shift through exchange of ideas. In the context of this study, this
refers to how clinician-clinician and clinician-patient communication affect the un-
derstanding of risk in the patient. Healthcare is very much a human-centred do-
main where interactions affect the outcome. Sharma (2006) shows how theories

17. 13
can be reconciled to provide a richer understanding of sensemaking. Interper-
sonal interactions help the sense maker notice residue, and consequently change
schemas. In addition, patients themselves can be the information source and col-
laborate with the clinician in the data coverage loop. This suggests that rather
than simply interpreting newly discovered data, sensemaking is about creation
and invention between the various actors.
Klein, Phillips,
Rall, & Peluso
(2007) propose
a sensemaking
theory, which
successfully
condenses the
characteristics of
the previously
discussed models.
Data-frame theory (Figure 2)
states that the sense maker
places data into a frame about
what that data represents; pre-existing
frames (results of previous experiences) influence how the new data is framed.
Three cycles make up the process of sensemaking; elaborating, preserving and
reframing. There are a total of seven steps in the data-frame model:
 Data and frame connection –data set is connected to a frame.
 Questioning the frame – unexpected or surprising data is encountered and a
frame is questioned.
 Elaborating a frame – a frame is elaborated but not changed due to new data.
 Preserving the frame – data is disregarded or ignored, preserving the frame.
 Seeking a frame – recalling or constructing a fitting frame.
 Comparing multiple frames – numerous frames are compared.
 Reframing – a frame is either replaced or combined with another.
Figure 2 Data-frame theory of sensemaking.
Redrawn from Klein, Phillips, Rall,
& Peluso (2007).

18. 14
Regardless of whether we talk about schemas, or frames, they both refer to the
way that individuals subjectively look at, filter, and sort the data that they encoun-
ter. A number of questions can be raised when making sense of how clinicians
think about risk in a patient. How do clinicians deal with inconsistencies and
anomalies in data? How do they judge the plausibility and quality of data? When
do clinicians seek and infer or disregard data? How do they seek and infer new
relationships in data? In later chapters, we will attempt to explain our findings
through the sensemaking lens by using the data-frame theory as a framework for
understanding how clinicians think about complex data and risk.
As we can see from exploring these models, sensemaking is a cyclical and itera-
tive process where data is collected and assimilated into pre-existing frames, or
frames are modified based on previous experiences. Sensemaking is about gen-
erating new internal frameworks based on new data. In the context of this thesis,
these sensemaking models provide a way to explain how clinicians deal with the
complex data that they are presented with. The visualisations created for this the-
sis are informed by sensemaking literature and focus on supporting the pro-
cesses clinicians go through when making sense of risk.
There are a number of existing applications of visualisation techniques in
healthcare. Rind et al. (2013) explore effective ways of visualising electronic
health records. Bui & Hsu (2010) discuss systems for adaptive visual interfaces
that integrate clinical information necessary to users’ aims. Faiola & Hillier (2006)
show how complex clinical datasets can be transformed into contextual
knowledge using visualisations, improving the quality of clinical decision-making
and decreasing the time wasted foraging for information by organising it in a con-
text-related format in a single location. Others have looked at the applications of
visualisation for classification and assessment of risk in chronic heart disease
(Harle, Neill, & Padman, 2012) and diabetes (Harle, Neill, & Padman, 2008).
However, when it comes to research into visualisation for healthcare, few studies
look through the sensemaking lens.

19. 15
Faisal, Blandford, & Potts (2013) identify potential ways that information visuali-
sation can assist both clinicians and patients in making sense of health data, but
conclude that more work needs to be done in order to incorporate the sensemak-
ing processes into the design of these tools.
Each person makes sense in his or her own way; sensemaking does not occur
externally, but by definition, inside the mind of the user. As users engage with vis-
ual representations, they also interact with the interface itself, in order to do so
they rely on the mental model that they develop (Sarah Faisal, Cairns, & Bland-
ford, 2007). This internal creation of concepts happens through interaction with
the external world.
Kirsh (2010) identifies a number of ways external representations help sense-
making and allow us to ‘think more powerfully’. As described in the cyclical pro-
cesses of sensemaking theory, when one experiences externalisations, the inter-
nal conceptualisations of a domain are generated, updated and used (Russell et
al., 1993). These externalisations can be in the form of visualisations. Spence
(2007) recognises visualisation as a cognitive activity; when designed well, visu-
alisations can amplify cognition and in turn amplify the sensemaking process
(Card et al., 1999).
Card et al. (1999) state that the purpose of visualisation is insight, as opposed to
just being ‘pictures’ to look at. Insights can be gained when data is represented in
a visual manner, thus supporting the user through visual sense making. Pirolli &
Card (2005) talk about insight being engrained in the sensemaking tasks; infor-
mation gathering, re-representation of data in schema, creation of insight through
manipulating representations, creating a knowledge product or direct action. In-
sight is but a single step in the sensemaking process, but sensemaking may not
be the only way to gain insight (Yi, Kang, Stasko, & Jacko, 2008).

20. 16
Yi, Kang, Stasko, & Jacko (2008) propose four ways in which users gain insight
through information visualisation. Firstly, provide overview is about understanding
the big picture; it informs the user of what is known and what is not known about
a data set. Although it is not directly related to gaining insight, it leads to an un-
derstanding of what parts need further investigation. Adjust is about changing the
level of abstraction or range of selection, this can be done by filtering or grouping.
Detect pattern is about finding trends, relationships, outliers etc. During this pro-
cess users may not only discover what they were looking for but also discover the
unexpected. Match mental model is about decreasing the gap between the data
and the mental model (Johnson-Laird, 1983) of the user, thus reducing cognitive
load.
These processes are not separate and can be used together to gain insight, they
are cyclical and iterative, much like sensemaking. These processes are relevant
to the design of visualisations in this thesis; however, we found no concrete
guidelines in the literature for designing visualisations to provide users with in-
sight. Insight is a qualitative process (Saraiya, North, Lam, & Duca, 2006) making
it well suited to exploration with the methods used in this study.
The previous sections in this chapter focused on sensemaking theory and the
way in which visualisations can move away from simply communicating known
insights in the data toward an exploratory process of iterative understanding that
supports the sensemaking process.
This section will give an overview of information visualisation principles and ap-
propriate concepts from pedagogues of information visualisation (e.g. Mazza,
2004; Spence, 2014) that were used in the design of visualisation in this thesis.

21. 17
The data that clinicians deal with comes from plethora of sources; verbally re-
ported data, sensors, health monitors, clinical tests etc. Card et al. (1999) men-
tion a number of points to consider before information visualisation visual repre-
sentations of data can be made:
 Data measurements – Nominal data is categorically discrete data such as
(e.g. behavioural, genetic, social, demographic). Ordinal data has a natural
ordering but the intervals between values are not the same (e.g. high, me-
dium, low risk). Interval data is numerical data (e.g. integers or real numbers).
 Data dimensions – univariate (1 dimension), bivariate (2 dimensions), trivari-
ate (3 dimensions), and multivariate (4 or more dimensions).
 Data structure – linear (made up of arrays, tables, lists etc.) temporal, spatial
or geographic (maps), hierarchical (taxonomies, genealogies etc.), network
(graph structures).
 Interaction type – static (print), transformable (user can manipulate)
Visualisations can be an effective way of representing information if designed
well. People assimilate information much more rapidly through visualisations than
they do through text (Ware, 2013). This section will cover the ways in which vis-
ual elements can be used to facilitate this.
Visual variables create mappings and structures; these should pull out interesting
features from the data. It is possible to take advantage of pre-attentive pro-
cessing to design effective visualisations. In this case, defined as the term as-
signed to objects that are processed faster than 10ms (Treisman, 1998):
 Form – line direction, size, curvature, grouping, marking, and luminosity
 Colour – hue and intensity
 Motion – flicker and direction of motion
 Spatial position – position, stereo-depth, convexity and concavity

22. 18
Figure 3 Visual types. Image by
Krygier & Wood (2005).
Bertin (2010) identified attributes
that he called ‘retinal variables’
in his 1967 work, Sémiologie
Graphique. Each of these
variables were identified as best
used to show either or both
quantitative and qualitative data.
Krygier & Wood (2005) expanded
on these ‘retinal variables’ – size,
colour value, texture, orientation,
and shape – by representing
them in points, lines, and
areas (Figure 3).
Visual properties refer to the way in which we are able to create differentiation in
the visualisation and effectively show representations. Fry (2004) identifies con-
trast as the most fundamental visual property. Gestalt principles (Wagemans et
al., 2012) explain how we notice visual elements as being contrasting. Pre-atten-
tive features are all ways to differentiate or contrast visual elements. Hierarchy is
about the order of importance of elements; visualisations should emphasise ele-
ments important to the task and de-emphasise those that are not, this can be
achieved through creating a hierarchy. Grouping is about clustering elements to
imply a relationship or shared meaning. Grouping creates patterns; dissimilar ele-
ments that are grouped together can also highlight differences or contrast.
Weight of elements such as the size or thickness of lines can show relative im-
portance or differentiation. Prominence should always be on ‘showing the data’
(Tufte, 1995), distracting with design runs the risk of data representations being
missed. Use of borders must be carefully thought out as not to increase the
amount of ‘non-data ink’.

23. 19
Figure 4 Hue,
value, and
saturation.
On a screen, colour is represented by a combination of red, green, and blue.
When referring to colour, the model of hue, brightness, and value
is better understood by the human mind (Figure 4). The hue
is what would usually be meant when colour
is mentioned (for example green or
magenta) value is the range
of black to white, and
saturation is the intensity
of the colour. Colour is useful
for contrast and mapping data.
Placement conveys hierarchy by ordering elements. Contrast can be shown
when an outlier is placed away from a group of similar elements. Grouping is the
principal use of placement.
A problem with displaying complex data is that it cannot be easily displayed in
one view. The user must be able to transform the view in order to use externali-
sations to forage for information.
A number of taxonomies have been proposed by researchers such as ‘overview,
zoom, filter, details-on-demand, relate, history and extract’ from Shneiderman
(1996), and ‘zoom, pan, scroll, focus+context and magic lens’ by Spence (2014).
These taxonomies describe low-level interaction techniques. We refer to the in-
teraction techniques proposed by Shneiderman (1996) when we talk about con-
crete operations in the visualisation that do not imply the cognitive aspect of user
intent.

24. 20
Yi, ah Kang, Stasko, & Jacko (2007) present a taxonomy based on user intent, or
what the user aims to do by interacting with the system, thus adding a cognitive
dimension to interaction. In this thesis, we adopt this taxonomy to refer to user in-
tent and the user tasks the visualisations were intended to support. The following
are the seven user intent interaction techniques proposed by Yi, ah Kang,
Stasko, & Jacko (2007):
 Select – mark something as interesting
 Explore – show me something else
 Reconfigure – show me a different arrangement
 Encode – show me a different representation
 Abstract/Elaborate – show me more or less detail
 Filter – show me something conditionally
 Connect – show me related items’
Armed with a toolbox of sensemaking and information visualisation knowledge
we can move toward creating a tool that supports the clinician in understanding
risk in a patient.

25. 21
3 METHODS
This chapter describes the participants and recruitment, apparatus and materials,
data gathering methods, and ethical considerations of the study.
The study was comprised of a total sample of 10 participants. Eight of whom
were male and two were female. All participants were doctors from primary or
secondary care. Participants were either general practitioners or specialists. Nine
participants were from the UK and one participant was from the USA.
The study was divided into three stages; six participants took part in Stage 2 and
ten participants took part in Stage 3 (including the six from Stage 2). Sessions
were performed face-to-face (F2F) where possible, but some sessions had to be
performed remotely for pragmatic reasons.
Participant summary:
 Number of participants: 10 (8 male, 2 female)
 Inclusion criteria: clinicians in primary or secondary healthcare who need
to about ‘risk’ of developing diseases when dealing with patients
 Demographic: 9 United Kingdom, 1 United States of America
Purposive sampling (Jupp, 2006), a form of non-probability sampling, was the
main method for recruitment. A variety of specialists and general practitioners
were selected as the sample of they matched the inclusion criteria. A range of
specialists and general practitioners were chosen in order to gain an understand-
ing of the similarities and differences in the way various clinicians think about risk.
Recruitment was carried out through email and word of mouth using industry and
academic connections.

26. 22
Figure 5 (A) Lenovo Thinkpad
with Stage 1 Visualisation 1 on
screen. (B) Samsung 700T
with Stage 2 visualisation
on screen.
The following software, hardware, data gathering tools, printouts & documents
were used in the study:
 Adobe Creative Cloud for creating designs; Stage 1 & Stage 2
 Axure RP for creating the Stage 3 visualisation
 Camtasia Studio® for capturing audio and screen activity
 Voice recorder as a back-up for capturing audio
 Lenovo Thinkpad 3680K16 laptop, Windows 8 64-bit, Intel® Core™ i5 CPU
M540 2.53GHz, 4096MB RAM, 1280 x 800 resolution, seen in Figure 5(A)
 Samsung 700T tablet, 1366 x 768, seen in Figure 5(B)
 10 x Participant information (Appendix A)
 10 x Consent form (Appendix B)
 2 x Interview plan: 1 x Stage 2 (Appendix C) & 1 x Stage 3 (Appendix D)
 Notebook for taking notes during interviews and think-aloud
 Pink and yellow Post-it® notes for creating the affinity diagram
 Coloured pens for colour-coding the affinity diagram
A B

27. 23
A semi-structured interview (Gillham, 2005) format was used – rather than struc-
tured or open interviews – in order to strike a balance between structure and
openness. This enabled a wide scope of questioning whilst remaining on topic.
This method was useful for both gathering requirements for re-design of the visu-
alisations and understanding how clinicians think about risk using complex data.
Semi-structured interviews made it possible to cover important questions while
also allowing for the pursuit of unanticipated themes as they arose. Interview
guides were used to guide the researcher’s line of questioning. Audio recording
of the semi-structured interviews was used to transcribe the interviews.
The purpose of the initial part of the interview was to make the participant com-
fortable, and learn about the clinical work they are involved in. This was useful to
understand the context of the problems they face, since participants came from a
variety of specialisations and medical fields. The middle of the interview was ded-
icated to constructing a deep understanding of participants’ work and sensemak-
ing activities. The aim of this part of the interview was to understand how clini-
cians make sense of complex health data in order to assess risk in a patient,
leading to a better informed understanding of how it might be possible to design
tools to support this. The end of the interview was used to bring up any lingering
points that the participants felt had not been covered.
Focus was always on framing the questions in real-life incidents that the partici-
pants had encountered, they were encouraged to talk about specific incidents ra-
ther than the general. The interviews were also a good way of identifying possible
biases within the sample (e.g. differences between medical fields), helping to mit-
igate those biases when analysing the data.
Nevertheless, interviews do not always elicit all interesting information from par-
ticipants; things that are obvious to the participant but not to the researcher may
be overlooked and remain unmentioned, therefore it was also beneficial to use

28. 24
think-aloud protocol. Conversely, participants will not mention every part of their
thinking during the think-aloud either because they do not think it is important, or
because they are not consciously aware of the particulars of their thought pro-
cesses. Interviews are a good way to extract the information that might be omit-
ted in the think-aloud session.
Think-aloud protocol (Boren & Ramey, 2000) was used in Stages 2 and 3 of the
study, whereby participants verbalised their thoughts as they completed a task.
Verbalising their thoughts helped to highlight differences in the user’s mental
model and the system image. Think-aloud data contributed to the iterative devel-
opment of the visualisations used in the study by fixing the limitations of the pre-
ceding designs (Ericsson & Simon, 1993).
Before the session started, participants were provided with detailed instructions
of how to think aloud and were encouraged to speak freely as they noticed things
in the visualisation. Whenever participants stopped thinking aloud, due to becom-
ing too involved in the task or forgetting to keep reporting verbally, they were
prompted (by the moderator) to continue. Care was taken to remain sensitive to
when the participant needed prompting, to prevent interruptions in the partici-
pants’ thought processes.
Unnecessary questioning was avoided, as users who are asked for information
about something they are not attending to in the think-aloud are forced to infer ra-
ther than recall their mental processes (Anders & Simon, 1980), leading to inac-
curate reporting.
The advantage of having participants verbalise their thoughts over merely ob-
serving their activity, was that it enabled the articulation of their understanding of
the activity.

29. 25
A risk assessment was completed and ethical approval for this study was gained
through University College London Interaction Centre.
Informed consent was given by all participants after they read the participant
information and signed the consent form. The participant kept one signed copy of
the consent form and the researcher kept another. Participants were all healthy
adults and did not belong to vulnerable or dependent groups.
The study adheres to the Data Protection Act 1998. Data was gathered with
consent, kept confidentially and securely. All participant data was anonymised
and made unidentifiable in reports and other shared materials.

30. 26

31. 27
4 DESIGN, STUDY & ANALYSIS
This chapter presents a chronological account of the steps taken to
generate the results of this thesis. We describe the details of the ob-
jectives, study design used with participants, visualisation design,
analysis, and outcomes of each stage. The motivations behind the
approach taken, as well as reflections on the strengths and weak-
nesses of those approaches are also presented throughout.
To begin with, we cover the development of the patient personas for
use in the visualisations. Then, the three stages of the study are pre-
sented. Firstly, the iterative design and evaluation of the two visuali-
sations in Stage 1 are described. Secondly, we explain the parallel
design approach to the two Stage 2 visualisations, the interview and
think-aloud procedure with participants 1-6, and then Stage 2 analysis
& requirements generation. Thirdly, an account of the final design of
the single Stage 3 visualisation, the study protocol with participants 1-
10, and Stage 3 analysis is given.
Three patient personas were created for the purposes of the study.
Unique patient information was generated for each of them. These
personas were not based on real patients or real patient data, the pa-
tient persona information was synthesised from research into particu-
lar conditions.
The personas were created with characteristics that would not clearly
place them at exceptionally high or low risk of a condition, this was
done in the hope that it would tease out the way in which participants
thought about risk.

32. 28
Figure 6 - Patient
personas for use
in the think aloud
scenario and to
populate the tool
with data.
In order to verify the personas, we consulted with a nurse in the
healthcare industry to review them; the feedback was used to update
and improve the persona data to be more representative of a ‘typical
patient’.
Persona A was Diego Blanco, a 35-year-old male with potential type
2 diabetes risk. Persona B was Deirdre Maguire, a 64-year-old female
with potential melanoma risk. Persona C was John Smith, a 75-year-
old male with potential lung cancer risk. These personas had thirty or
more pieces of patient information each. Each piece of patient infor-
mation (ethnicity, BMI, diet etc.) was placed into a category (social
history, clinical stats, behavioural etc.). The information was then as-
signed a value (Hispanic, 29, high calorie etc.), metadata (eats out
with clients, low activity due to family life etc.), and risk severity (gen-
eral patient information, reduced risk, low risk, high risk etc.). The pro-
file photographs and information for patient personas can be seen in
Figure 6.
There were two uses for the personas; first, the data set of each per-
sona was used to populate the visualisations with data, and second,
the personas were used in the think-aloud sessions to introduce the
scenario and the task where participants evaluated a patient persona
for risk of developing a specific condition in the near or distant future.

33. 29
The aim of Stage 1 was to use the persona data to produce a concept
design that informed the visualisations in Stage 2. An iterative ap-
proach (Nielsen, 1993) was adopted to improve the Stage 1 visualisa-
tions. Two visualisations were created in Stage 1. Multipage PDF
documents were created as the artefacts for each of the two visuali-
sations in Stage 1. Each page of the PDF had one screen of the inter-
face on. Both Stage 1 visualisations relied on existing information vis-
ualisation literature (Section 2.2.2) as a reference for design. Expert
evaluation with an information visualisation professional was used to
evaluate and provide recommendations for the next iteration.
The first Stage 1 visualisation consisted of two views; Compound
View and Category View. A risk severity number on an interval scale
of 1-8 was assigned to each risk factor; the designer assigned this ar-
bitrarily, but it can assumed that real software would use relative risk
from population studies to assign severity. Patient information was
plotted in circles along the x-axis according to their risk severity. A
search bar was present in both views to perform a query on the da-
taset to reduce the amount of data that is visible on screen. For ex-
ample, filtering by category or individual risk factor. Clicking on an in-
dividual circle would bring up information associated with that piece of
patient data, thus providing details-on-demand to the user upon re-
quest.
The Compound View, illustrated in Figure 7(A), gave an overview of
the whole dataset. In this view, the circles were pushed outward on
the y-axis subject to the amount of other circles already in that area,
making the concentration of circles larger with the intention of making

34. 30
Figure 7 Stage 1 iteration 1.
(A) Compound View.
(B) Category View.
the distribution of risk more apparent pre-attentively. An overall risk
value was assigned based on the average distribution of circles. The
circle colour was mapped to patient information categories. A different
set of colours was also assigned to risk factors that were changeable
(can change) and risk factors that that were not changeable (can not
change) through intervention; a key was placed in the bottom right
hand side to act as a reference.
In the Category View, illustrated in Figure 7(B), the categories (ge-
netic, medical history etc.) were divided along the y-axis and as-
signed to lines along the x-axis. The average risk for categories was
displayed, showing the distribution of risk within individual categories.

35. 31
The second Stage 1 visualisation (Figure 8) mapped changeable/non-
changeable to shapes in order to decrease the amount of colours
used; circles showed ‘can change’ whilst triangles showed ‘can not
change’. The 1-8 interval scale was divided into an ordinal scale of
protective, neutral, and low, medium, med/high and high risk. Filters
were added to show what the patient could change and what the clini-
cian could change through intervention, allowing the user to adjust.
Figure 8 Stage 1 iteration 2.
Category names and colours
updated from previous version.
Filters added to search types
of data. Triangles and circles
differentiate between ‘can’ and
‘can not change’. Risk has
been split into protective,
neutral & low-high relative risk
for each data point.
(A) Compound View.
(B) Category View.

36. 32
By filtering, the user was able to see something conditionally, for ex-
ample ‘show me things that the patient ‘can change’ that have been
proven to influence the risk of type 2 diabetes’.
Expert evaluation was practical because it could be done at any time
and with minimal resources, providing a satisfactory cost-benefit ratio
(Nielsen, 1994). In contrast, issues can be missed (validity) and differ-
ent experts can find different issues (reliability). A data visualisation
expert was consulted and a ‘simplified think-aloud’ was carried out to
identify and provide suggestions for re-designs. This type of evalua-
tion is no replacement for real users; this method was used in order to
resolve basic usability and design issues before sessions with real
participants, so that the focus of testing would be on how the visuali-
sation supports clinicians’ sensemaking. Below are some examples of
the issues that were identified through the expert evaluation:
Categories are a nominal type of data. In the visualisations, the mis-
take of representing the data in a way that implicitly suggested an or-
der to it was made. The colour choices in both Stage 1 visualisations
were sequential in nature, using diverging schemes of colour led to
confusion. Using progressive variations transitioning between hues
suggested continuity, something that is not present in the categorical
or nominal data (Silva, Sousa Santos, & Madeira, 2011).
Since we assumed that actionable data is what makes clinical inter-
vention possible, we wanted to see if visualising the factors that could
be changed through intervention would support clinicians thinking
about risk. The first visualisation used colour to differentiate changea-
ble factors, while the second visualisation used shape to represent
the same thing. Although the latter was more effective, it was not use-
ful for clearly seeing how the risk is weighted among changeable and
non-changeable factors.

37. 33
Table 1 Visualisations and patient
personas matrix showing the six
variations.
Horsky et al. (2012) state that ‘poor usability is one of the core barri-
ers to adoption and a deterrent’ to use of clinical decision support
systems. Upon reflection, using the iterations in Stage 1 to identify
and fix usability and design problems provided a good foundation for
designing better visualisations in the sessions with real participants.
Following Stage 1 visualisations, work begun on Stage 2 visualisa-
tions where a parallel design approach (Nielsen & Faber, 1996) was
used. The rationale behind using parallel design was that less time
was required to explore designs than if they were produced sequen-
tially. Parallel design was useful for testing and comparing visualisa-
tion types, presenting the same data set in different visual structures.
Two designers worked simultaneously in this stage; each worked in-
dependently on different visualisations. The Principal Investigator
worked on Visualisation 1 and a different designer worked on Visuali-
sation 2. Two separate designs were created for Stage 2; the three
personas and their data were used to populate each of the designs.
As before, the designs were exported to multipage PDFs, this time
the PDF documents were created with interactivity in the form of click-
able parts of the interface that were hyperlinked to other parts of the
document. A total of six variations were created; these can be seen in
Table 1. The artefacts created in Stage 2 were used on a laptop with
participants during the think-aloud part of the study protocol.
Persona A Persona B Persona C
Visualisation 1 1A 1B 1C
Visualisation 2 2A 2B 2C

38. 34
In Visualisation 1, illustrated in Figure 9, the colour scheme for cate-
gories was changed in an attempt to avoid implying magnitude differ-
ences between categories. Differences in hue with only slight differ-
ences in the lightness were used to differentiate categories, but using
nine colours to represent the categorical data made it hard to discrim-
inate between categories. As a consequence, the ability for the user
to memorise the meaning of each block in the visualisation was di-
minished, MacDonald (1999) suggests using seven or less colours to
show data of this kind. In Stage 2 Visualisations, the name of the per-
sona was added to the title, along with the condition they were sus-
pected of being at risk of. In addition, a frame at the bottom-centre of
the screen was added with the patient name, photograph and dummy
text for notes on the patient.
Figure 9 Stage 2
Visualisation 1 with
Compound View
selected.

39. 35
Figure 10 Compound View
with ‘early osteoporosis’
selected in the main
visualisation, thus
changing the bottom-
centre frame content to an
image of the patient’s
DEXA scan with notes.
The content of this frame changed as the visualisation was interacted
with, as can be seen in the Compound View (Figure 10). The selected
block was highlighted with an orange outline and a line connected the
selected block to the bottom-centre frame, inferring a relationship. We
also included a maximise button in the top right hand of the frame to
expand the window and zoom into the data subset. This interactivity
was also present in in the Category View, illustrated in Figure 11.
In the Compound View, each item of patient data was represented by
a solid block of a fixed size. These blocks stacked upon each other
additively in columns out from the central line that divided factors that
‘can’ and ‘can not change’. This allowed spatial grouping to be used,
instead of colour or shape, to differentiate between the entities ‘can
change’ and ‘can not change’. This made it possible to distinguish the

40. 36
Figure 11 Category
View with ‘nevi checked’
selected, notes and an
image of the patient’s
nevi is displayed in the
bottom-centre frame.
weighting of risk in each risk severity group through the height of the
block stacks from the distance the stacks protruded from the
mid-section.
In the Category view, ‘can change’ and ‘cannot change’ were still rep-
resented with shape. However, instead of using triangles and circles,
as was the case in the second Stage 1 visualisation, squares repre-
sented ‘cannot change’ and squares with rounded edges represented
‘can change’. The user is able to encode the data in a different repre-
sentation by shifting views from Compound view to Category view
and vice versa.

41. 37
Figure 12 Visualisation 2.
Three columns (left to
right) show factors that
reduce risk, patient
information and factors
that increase risk.
In Stage 2 Visualisation 2, illustrated in Figure 12, three separate col-
umns were used to visualise data. The central column contained the
total information that was available for that patient. The left column
represented factors that reduce risk and the right column represented
factors that implied increased risk for the condition being assessed.
In Visualisation 1, the severity of risk of an individual data point
relative to a condition was shown by separating data into columns;
low, medium and high risk. In Visualisation 2, the severity of risk was
represented by the size of a block in the ‘Increase Risk’ column. Lines
were drawn from the central ‘Patient Information column to show and
infer relationships between factors that reduced, increased, or had no
direct correlation to risk.

42. 38
Figure 13 Individual data point (A) Exposure to
radon. Map from ukradon.org (B) Lung X-rays.
Clicking on a data point would enlarge the
patient information bar and move it to the
left hand side of the screen and let the
user look closer at the data subset for that
patient data (Figure 13); what Tufte (1995)
would refer to as a micro view or what
Shneiderman (1996) refers to as zoom.
Zooming added deeper contextual text to
the enlarged patient information column
for each data point. A red line on the right
side of a data point in the patient
information column meant that it was a
risk factor, while a green line on the left of
the data point meant it was a risk reducing
factor.
The additional properties related to a data
point were displayed on the right hand
side of the zoomed in patient information
bar. Some of the screens displaying
zoomed in patient information, such as in
Figure 13(A), had the design error of
heavy use of thick lines as borders in the
design placing less prominence on
‘showing the data’ (Tufte, 1995). This led
to users becoming distracted with the
design rather than with the actual data the
design was trying to display; excluding
this from the design would have reduced
the amount of ‘non-data ink’.
A
B

43. 39
In advance of the actual study, the protocol was piloted with a USA
based doctor in the healthcare industry in order to modify and im-
prove interview questions and think-aloud procedures. The feedback
from the pilot session has not been included in the sample. Prior to
conducting the sessions, participant information and consent form
documents were sent in an email to participants 1-6 (Table 2). The
body of the email confirmed the time and place of the interview (con-
ference call details if session was remote).
Upon commencement of the session, the Principal Investigator – ac-
companied by his Industry Supervisor – asked participants if they had
any questions about the study and if they understood everything in
the participant information. Upon confirmation that the terms of the
study were understood, consent forms were signed and collected (a
signed and scanned version of consent form for remote sessions).
Participants were given a short introduction to the procedure that
would follow; firstly, an interview involving current workflow, decision
making processes, the actionability of data, how they make sense of
the information they deal with, the various sources of data, trustwor-
thiness of data and communicating risk to others. Secondly, a think-
aloud session that would involve looking at two different visualisa-
tions, where the participants would explore a patients risk of develop-
ing a condition.
The researcher then started audio recording on the voice recorder,
and audio & screen recording on the laptop using Camtasia Studio®.
The researcher also took notes using a notebook and pen during both
interview and think-aloud.
The semi-structured interview, described in Section 3.2.1.1, took
place. The researcher asked the main questions and followed up with

44. 40
Table 2 The column ‘Vis’ shows the visualisation order and patient persona used during the think-aloud. For
example, participant 3 saw visualisation 1 with patient persona A followed by visualisation 2 with persona C.
additional questions to probe further. Clarifying questions were asked
when a point was unclear or confirm of what a participant meant want
needed. The interview took around 30 minutes per participant.
Once the interview part of the session was over, the researcher ex-
plained think-aloud protocol (Section 3.2.1.2). Participants were told
that they would explore two visualisations representing complex
health data about a patient, and were asked to talk about what they
understood from the interface and how it might relate to their task of
assessing risk. The scenario was explained; the hypothetical patient
in question was sitting outside in the waiting room and that this was
the first time that the participant was viewing the patient’s data. Their
task was to discern whether the patient was at risk of developing a
condition based on what was understood from the interface. The
think-aloud session took around 30 minutes per participant. Around
15 minutes was spent thinking aloud about each visualisation.
Upon completing the interview and think-aloud the session was con-
cluded; participants were de-briefed and asked if they would be will-
ing to be re-contacted for Stage 3 of the study. Once all six Stage 2
sessions had been completed, analysis took place.
Participant Gender Medical field Location Vis F2F
P1 M Cardiology UK V1A, V2A Yes
P2 M Cancer genomics USA V2B, V1A No
P3 M Clinical Pharmacology & General Medicine UK V1A, V2C Yes
P4 M Psychiatry UK V2B, V1B No
P5 M General Practitioner UK V1B, V2C Yes
P6 M Paediatric Pathology UK V2C, V1C Yes

45. 41
This section describes the approach to analysis, findings from the
Stage 2 study with participants 1-6, and requirements statements.
The main purpose of analysing data after the Stage 2 study was to in-
form the design of the Stage 3 visualisation. However, findings from
Stage 2 were also integrated into results in Sections 5.1-5.4.
To begin Stage 2 analysis, all interview and think-aloud data was
transcribed word-for-word using the audio from the voice recorder.
When it was not clear which part of the interface a participant was
talking about in the think-aloud recordings, the screen recording was
used as a reference.
Interviews were annotated in a word processor with approximate
codes. Recurring patterns in the way clinicians think about risk, prob-
lems they face, work practices, work environment, and attitudes to-
ward data helped to form initial interpretations.
The majority of analysis in Stage 2 was on the think-aloud data. The
think-aloud transcripts were printed onto A4 paper and a highlighter
was used to mark substantive statements for each participant.
Post-it® notes were then attached to these substantive statements,
the theme of the statement was summarised and an identifying code
was written on the Post-it® to recognise where the data came from in
the transcript. The letters and numbers in square brackets in Section
4.3.5.1 follow the same order as these codes; participant number, vis-
ualisation number, patient persona. For ease of visual differentiation,
Pink Post-it® notes were used for visualisation 1 and yellow Post-it®
notes were used for yellow visualisation 2.

46. 42
The following pattern was used to identify where data came from:
Once all of the Post-it® noted has been added to the transcript high-
lights, an affinity diagram (Hartson & Pyla, 2012) was used to aid
analysis (Figure 14).
Large sheets of A1 paper were taped together and the Post-it® notes
were taken from the transcripts, placed in clusters with similar
themes, and given topical labels on the A1 paper. These clusters
soon formed groups within hierarchies; these were all labelled. Even-
tually a structure began to emerge.
Figure 14 Affinity
diagram created
during analysis.
42

47. 43
Through the creation of the affinity diagram, both interface specific
observations and abstract findings about clinicians needs began to
emerge:
Although participants said Visualisation 1 was a tool with a structured
presentation [P1V1A], they had to deal with various data at the same
time [P1V1A].
There were also usability issues; for example, a participant was con-
fused about the horizontal relationship of data points in the Com-
pound View, even though there was no meaningful relationship in-
tended in the design of the visualisation [P4V1B].
It was noted that this tool might be good for a specialist consultation
[P1V1A] rather than for a busy general practitioner, since it took a
while to digest all of the information on the screen.
43

48. 44
Visualisation 1 had too much data to process at once, leading to par-
ticipants asking for more ‘black and white’ and ‘yes and no’ [P1V1A].
In Visualisation 2, participants said that the interface was putting all
variables for assessment in one place [P6V2C], which gave a nice
sense of the data collected and how it influences risk up or down
[P2V1B]. The simpler information structure of the overview in Visuali-
sation 2 helped participants to understand the three columns
[P2V2B]. However, it was not always apparent that the columns were
not equivalent; the size of the data points relating to severity of risk
was not a connection that was easily made [P4B2B], also having a
smaller pixel area for low risk factors made interaction a challenge
[P1V2A].
In both visualisations, users generally had trouble interpreting the sig-
nificance or meaning of category colours [P6V1C] [P6V2C] [P1V2A]
[P4V2B]. This led us to believe that using colour to differentiate be-
tween categories is not essential to the task of assessing risk. Using
colour that does not provide further insight for the user can be per-
plexing as they try to understand its meaning and, hence, should be
avoided (MacDonald, 1999).
The filters in both visualisations were not visually apparent to all users
[P4V2B] and the terminology was not well understood [P3V2C]. The
readability of Visualisation 2 was generally better than Visualisation 1;
only one participant noted the readability of the text was poor in Visu-
alisation 2 [P4V2B], in Visualisation 1 participants complained about
small boxes, end-of-line hyphenation, differentiation between square
and rounded edge squares & small text size [P1V1A] [P2V1A].
Although both visualisations succeeded in showing complex data
from disparate sources in one interface, an understanding of whether
the patient was at risk or not was missing. Information about how a
single data point correlates to risk (why it is placed in high, medium,

49. 45
low or protective) [P6V2C] was not present. Adding context about
where a patient fits into a risk population would have helped.
Participants pointed out that an overall risk score was missing from
both visualisations, something the majority of participants noted
[P2V1B&2A] [P3V1A&2C] [P1V1A&2A] [P6V1C&2C]. Due to the vari-
ety of data represented in the visualisations, finding an existing risk
calculator or algorithm that considered all factors was not possible.
Nevertheless, including existing risk calculators to be applied on a
subset of data emerged as an option. In the end, risk is complex, but
in a clinician’s daily work, a summary is needed [P1V2A].
The think-aloud protocol used with the two Stage 2 visualisations
highlighted both strengths and weaknesses in their respective de-
signs. The interviews also revealed the way in which clinicians think
about trusting data, actionability, workflow, attitudes, understanding,
and communicating risk.
Following the completion of the affinity diagram, the designers dis-
cussed possible design ideas and any unanswered questions or
holes in the data that required further investigation in Stage 3. The
learnings were merged from both the think-aloud and semi-structured
interviews into a requirements statement document (Appendix E).
When moving from codes in the affinity diagram toward requirements,
focus was on matching the internal mental model of clinician through
externalisations. The rationale behind the requirements were dis-
cussed and design recommendations were made. Care was taken by
the designers to avoid bias toward their own competing design ideas
by looking objectively at the findings in the data when creating the
requirements document.

50. 46
Table 3 Requirement
Statement structure.
The requirements statements follow the structure shown in Table 3. A
traffic light metaphor was used to show priority, red (top) being high-
est priority and green (bottom) being lowest. Textures have been re-
dundantly mapped to the colours for colour-blind readers.
Priority was assigned requirements according to the severity of the
problem, which was derived from, to how frequently the topic or con-
cept in question came up in interviews and think-aloud data and the
amount of design work required to implement the change.
An assortment of requirements statements have been taken out of
the ‘Requirements Statement’ document (Appendix E) to show
as examples. These can be found in Table 4.
Requirement Statement Priority
#: Name of feature/category
Second-level feature/category
Requirement statement [place in Affinity Diagram]
Rationale (if useful): Rationale behind requirement
Design recommendation (optional): Commentary about requirement

51. 47
Table 4 Selected examples from
Requirement Statements
document
5. User input
Editing risk category
If there is no/low evidence for a data point, allow the user to assign a risk category. But, do not al-
low the strongly evidence based data points to be moved. Track all changes and show if an item
has been moved through the interface [information/supporting interpretation &
‘objective’ vs. ‘subjective’ data]
Rationale: ’Solid’ data has its limitations [P1V1A]. For example, occupational history is useful
[P5C1B], but interpretation is subjective. In clinical work evidence & subjective opinions are mixed,
this tool gives an objective view of both that can be reviewed [P6V1&2]
Design recommendation: The data points that aren’t used in validated scales can be re-assigned
to another risk category, but these changes must be tracked for later review
9. Individual data point view
Relative risk
Let the user know how a single data point correlates to risk [P6V2C] [risk type>relative risk]
Rationale: Relative risk is not known for all of the data points, but it would be helpful is it was avail-
able for those points that are known [P2V2B]
Design recommendation: Display relevant patient data on classic graphs & scales within the indi-
vidual data point view. Let user know the high/med/low in the overview is based on the relative risk
of that specific measurement in order to ground the perspective [P2V1A]
17. Overview
Overall risk calculation
The user needs an overall risk calculation that gives a quantitative measure [interface specific]
Rationale: Overall risk was missing from both visualisations, something the majority of participants
noted [P2V1&2] [P3V1&2] [P1V1&2] [P6V1&2]
Design recommendation: Individual conditions have their own risk calculators (i.e. risk of diabetes
in 5 years is X [P3V1A]). Include a risk calculator to display overall risk

52. 48
After defining the requirements, the design and implementation of the
Stage 3 Visualisation (also referred to as ‘the tool’) started. This in-
volved merging the best parts of the two versions produced in Stage
2 into one design.
In order to do this, the two designers used the requirements state-
ment document created from Stage 2 learnings. In Stage 2, the de-
signers worked separately, but in Stage 3, they worked collaboratively
on the visualisation.
The merged design aimed to apply findings about how clinicians think
about and make sense of risk in a patient by creating tool that re-
duced the gap between the clinicians’ mental model and the data in
the externalisations that they currently work with.
Before the design work begun, personas ‘A’ and ‘B’ were improved
based on feedback from participants in Stage 2. Persona ‘C’ was not
included in Stage 3 design due to temporal constraints in the develop-
ment period; this persona was discarded because it produced the
least interesting data during the think-aloud sessions.
Two artefacts were produced; one visualisation populated with per-
sona ‘A’ data and another with persona ‘B’ data. The Stage 3 visuali-
sation was created in the rapid prototyping tool, Axure RP, and then
exported to HTML and JavaScript for use on a Samsung 700T tablet
in Stage 3 of the study.

53. 49
Figure 15 The overview shows
the title bar, tool bar, patient
information, risk factors.
This section will discuss the design of each part of the tool and the ra-
tionale behind the design. The interface was interactive but some
parts such as ‘Go To Investigation’ were not functional. Data showed
that they were important to clinicians’ needs and understanding, but
beyond the scope of what could be implemented. They were included
in the interface to hint at what their functionality might be.
49

54. 50
The overview screen (Figure 15) was made up of a number of parts;
the header, patient information and risk factors. In Shneiderman's
(1996) mantra, ‘overview first, zoom and filter, then details-on-de-
mand’, overview refers to the act of looking at ‘the big picture’. The
overview screen in the tool does this by presenting all of the patient
data in a macro view (Tufte, 1995). Spence (2007) speaks of over-
view as the ‘qualitative aspect of some data’ – in this case the pa-
tient’s risk of developing a condition – that is ideally ‘acquired rapidly
and, even better, pre-attentively’. This screen attempted to achieve
this by showing the clinician factors that are known to contribute to
risk, how severe they are, and whether those factors can be changed.
Since user attention is first attracted to visually strong (big, colourful,
prominently placed) objects, the design attempted to lead users’ gaze
towards high risk factors. The user can then search for details among
less prominent elements in patient information. This was key because
if information is not organised in an optimal manner overview (Yi et
al., 2008) clinicians could potentially be stuck in the foraging loop (Pi-
rolli & Card, 2005) for longer than necessary.
The header, illustrated in (Figure 16), contained the title bar, serving
as a reminder of the purpose of the tool. Under that, basic patient in-
formation and photograph helping identification of the patient, useful
when a clinician deals with many patients. On the bottom, the condi-
tion the patient was being assessed as being at risk for, risk calcula-
tor selection, search bar, and buttons to lead onto next steps (which
were beyond the scope of this thesis) within the clinical workflow.
Figure 16 Header at the top of
the tool. Different conditions
could be selected from ‘Risk of
developing’ and risk calculators
can be applied from the
dropdown ‘Apply calculator’.

55. 51
Figure 18 Risk factors.
Figure 17 Patient Information.
Hierarchies and structures refer to elements within elements. They
also refer to an element that has a pointer to another element. In the
‘Patient Information’ (Figure 17) section of the overview the patient in-
formation was ordered into categories using lines to divide them. This
was designed in a structure that follows the clinical workflow that par-
ticipants described in Stage 2 interviews and think-aloud.
The coloured strips on the right hand side of the boxes relate to the
level of risk that piece of patient data is thought to have relative to an
empirical population study. All information available about a patient is
displayed here.
There is a danger that, if a part of the interface that contains less im-
portant data draws attention, features that are more important might
be overlooked. We tried to make sure that the most prominent fea-
tures of the interface were also the most significant parts of the data.
By placing all factors that are known to contribute directly to the risk
of a condition in one place (Figure 18) the clinician is able to conserve
mental resources that would otherwise be spent searching for risk
factors.
The location of elements also affects how a screen is viewed. For this
reason we grouped changeable factors separately from non-changea-
ble, indicating to the clinician which data is actionable. High risk fac-
tors are represented with highest saturation red and low risk factors
with the lowest saturation red.

56. 52
Figure 19 If the risk
calculator runs when
information is missing,
a prompt appears
requesting missing
information.
Selecting a risk calculator from ‘Apply Calculator’ would bring up an
overlay that superimposed a yellow hue on the factors that the calcu-
lator took into account. As a reference, the NHS (National Health Ser-
vice, 2013) calculator was used with the persona information to cre-
ate a more realistic outcome. If patient information that the calculator
algorithm needed was missing, as shown in (Figure 19), the tool dis-
played an alert prompting the clinician to gather that information.
52

57. 53
Figure 20 The completed
risk score calculation.
NHS Diabetes Risk
Calculator available from:
www.nhs.uk/Tools/Pages
/Diabetes.aspx.
The factors that were present and contributed to risk had a line drawn
across from ‘Patient Information’ to ‘Risk Factors’. By connecting two
elements, it is possible for a relationship between the two to be
shown. This feature was meant to allow the user to differentiate be-
tween factors taken into account by the calculator that did contribute
to risk and those that did not contribute to risk in the patient.
53

58. 54
Figure 21 Individual
data point view.
The advantage to overlaying the cal-
culator on top of the interface is that it
is plain to see which factors are iden-
tified as contributing to risk by other
evidence sources, but not taken into
account by that particular calculator.
Providing all the required information
was present, the risk calculator ran
and displayed an overall risk score
(Figure 20). This was the absolute
risk of developing a condition within a
timeframe. A segmented coloured
scale as well as descriptive text rep-
resented this.
Clicking on a piece of patient
information on the overview screen
would lead the user to an individual
data point view (Figure 21). This view
effectively zoomed into the patient
information bar, showing more detail
about each data point on the left and
providing extensive properties about
the patient information that was
selected. The individual data point
view displayed various types of
information depending on the type of
data; risk severity, relative risk (with
embedded visualisations to show it),
and contextual information.

59. 55
Figure 22 Examples of the
embedded visualisations
within the tool.
Individual data points had embedded visualisations in order to repre-
sent the patient data in a way that was easy to make sense of it. Only
data that was better represented visually was represented this way,
for example natural language notes were better displayed as textual
information. A number of the embedded visualisations are shown in
Figure 22. Examples of these representations included the following:
 Tables – these were used to show two-dimensional data. How-
ever tables are only useful when there are a limited number of en-
tries for the columns and rows, otherwise they get too crowded.
 Line graphs – these are a number of data points connected by
lines, showing continuity across the values. Line graphs were
used to show data such as hemoglobin A1c levels over time.
 Bar graph – were used to display series data where there
was no continuity between values. Bar graphs were used to show
data such as steps taken per day.
 Geographical map – these were used to show environmental ex-
posure and post code information.
 Matrix – wo dimensional sets such as the measures for BMI were
plotted in a matrix.
 Tree – used when hierarchically ordered data is used. This was
useful for showing family heart disease history.

60. 56
The bottom of the individual data point view had a series of buttons:
Notes – these were added to allow easy access
to notes when searching for information about a
data point or allowing the clinician to add notes
when they gain insight.
Guidelines – these were guidelines that link
directly to the National Institute for Health and
Care Excellence. This was added to support
clinicians through guidelines, advice and
appraisals.
Evidence – these were added for the times that
guidelines do not apply. Clinicians are able to
search for available evidence and latest literature
concerning a risk factor.

61. 57
More – there were three buttons in this submenu.
‘Go to Investigations’ and ‘Go to Diagnosis’ were
placeholder to hint at following phases in the clini-
cal workflow. As they were beyond the scope of
this work, they were not made interactive. Change
risk classification opened a pop-up window when
tapped.
Change risk classification – allowed the user to
flag a piece of patient information with a risk factor
(e.g. if the clinician finds out that the patient
spends too much time in the sun that contextual
factor can be flagged). Marking helps users make
sense of a domain on their own terms and track
their own developing understanding (Huang &
Eades, 2013).
Confirming the action – The risk classification
change is flagged for review if the user continues,
this acts as a safeguard for rogue actions. The
user is prompted before the change is submitted
for review; adding an extra step to confirm in an at-
tempt to minimise error.

62. 58
Following the completion of the tool, Stage 3 sessions with partici-
pants 1-10 were scheduled. Participants 7-10 were sent participant
information and consent form documents attached to an email, since
they had not taken part in Stage 2. Participants 1-6 simply confirmed
that they were still willing to take part in the study. Table 5 shows the
participants, order in which they saw the personas, and whether the
session was F2F or remote.
The protocol for Stage 3 sessions followed the same structure as
Stage 2 sessions, but with one main difference; the think-aloud was
carried out before the semi-structured interview. Sessions lasted
around 60 minutes, however this time the think-aloud sessions took
approximately 45 minutes and the interviews took approximately 15
minutes.
This time, when the think-aloud was explained, the participants were
asked to perform a number of tasks that were designed to test the as-
sumptions made about the visualisation being able to support clini-
cians thinking about risk:
 Use risk calculator to see a risk summary for this patient.
 Look through patient information and explore some of the data.
 Flag a piece of social information as a risk factor because you
know (persona specific scenario) might be putting them at risk.
Participants thought aloud with persona A and persona B. About half
of the 45 minutes was spent viewing the visualisation with each pa-
tient persona.
The interview plan (Appendix D) in Stage 3 sessions was also fo-
cused on evaluating the tool and the way complex health data was
visualised, as well as confirming preliminary evidence from Stage 2
about how clinicians make sense of risk in the patients they deal with.

63. 59
Participant Gender Medical field Location Vis F2F
P1 M Cardiology UK A, B Yes
P2 M Cancer genomics USA B, A No
P3 M Clinical Pharmacology & General Medicine UK B, A Yes
P4 M Psychiatry UK A, B Yes
P5 M General Practitioner UK A, B Yes
P6 M Paediatric Pathology UK A, B Yes
P7 F Ophthalmology (specializing in genetics) UK A, B Yes
P8 F Radiology UK B, A Yes
P9 M General Practitioner UK B, A Yes
P10 M General Practitioner UK B, A Yes
Table 5 Participants for Stage 3 study. ‘Vis’ shows the order that the patient personas were presented
to each participant during the think-aloud. For example, participant 10 saw the visualisation with
persona B followed by persona A.

64. 60
A thematic analysis (Braun & Clarke, 2006) was used to identify, ana-
lyse and report themes in the data. This approach was used to give a
rich description, reflecting the predominant themes that arose within
the data set. The multi-stage approach to the study meant that Stage
2 was focused on understanding users’ needs and practices to yield
requirements for Stage 3 design, but the overall aims were to under-
stand how clinicians make sense of risk. As was the case with Stage
2 analysis, interviews and think-aloud sessions from Stage 3 were
transcribed word-for-word by the researcher. This helped with becom-
ing familiar with the data and forming initial interpretations (Riessman,
1993).
Learnings from Stage 2 analysis were carried over to Stage 3. The in-
itial themes and approximate codes were reviewed as new data was
incorporated from the ten Stage 3 sessions. Notes made by the Prin-
cipal Researcher during the sessions was also used to support the
other sources of data. Data from research notes, interviews and think-
aloud sessions was triangulated (Guion, Diehl, & McDonald, 2011),
increasing validity of results.
At first, notebooks and paper were used when the focus was more on
the exploration of the data. Preliminary codes were first added to the
transcriptions, these codes were sorted by recurrent patterns in the
data. This process was done in an iterative manner, themes were cre-
ated for similar codes, and themes were merged and adapted in light
of new data. The main tool for analysis then shifted to a word proces-
sor, making it easier to organise data in a proper structure. This was
done systematically whilst building up a narrative about each theme,
reviewing themes when inconsistences in the data emerged. This
was done iteratively until a clear narrative emerged within the data.
The findings from this process are presented in the next chapter.

65. 61
5 RESULTS
This section contains the findings that arose from the data. Firstly, the main find-
ings are stated and discussed in Section 5.1. Sections 5.2 to 5.5 present and dis-
cuss secondary findings, which include the way in which clinicians systematically
organise their work, think about risk, arrive at a decision, and communicate risk.
Finally, Section 5.6 contains findings related to the tool in terms of its efficacy in
using visualisation to support clinicians dealing with complex health data.
The data collected from think-aloud and interviews and subsequent analysis
revealed two main findings:
I. Clinicians use a number of disparate sources of information to make sense of
risk in a patient – this entails complexity. There is a discrepancy between how
clinicians talk about their work, and what they actually do when assessing risk
in a patient.
II. We have promising preliminary evidence from this exploratory study that
tools, such as the one described in this thesis, can support clinicians by
creating externalisations that facilitate the implicit processes that they use
frequently in their work.
Clinicians routinely use complex data from a number of different places to try
to form a unified understanding of a patient’s risk. This diverse data is usually
presented in a way that makes the relationships between potentially significant
information difficult to perceive.
Our findings show that clinicians speak of the data that they use when assessing
risk as being ‘limited to validated clinical data’. They routinely use studies based
on specific populations and tools such as risk calculators. However, clinicians ac-
cept that research evidence is an abstracted generalisation that does not always
represent the actual risk of the individual patients they deal with. Patients do not

66. 62
always fit into the population of the epidemiological studies that they are com-
pared to. In reality, the patient may be found anywhere within the vast landscape
of risk. Similarly, guidelines are a common point of reference; however, they are
not always sufficient, as there are always outliers that do not fit the typical case.
The difficulty arises when it comes to making sense of what all of this actually
means for the patient that the clinician is attempting to assess.
Our participants described how they use studies, calculators, and guidelines in
conjunction with patient data related to risk of a certain condition, in order to as-
certain whether or not an individual is at risk and what – if any – intervention
should be taken.
Current health policy and evidence based medicine (Straus, 2011) state that clini-
cal practice is a scientific discipline where the science base is derived from ra-
tional, universal, and objective evidence. The rationalistic attitude of only utilising
validated empirical evidence to assess risk is not completely reflected in the way
that clinicians seek and use information in practice. We found that clinicians use
empirical tools and validated data alongside contextual factors specific to the pa-
tient they are dealing with when assessing risk.
In terms of data-frame theory (Klein et al., 2007), the empirical tools present data
that cause a connection to an existing frame about what the risk score or sum-
mary means. The clinician questions the frame by asking whether these
measures apply to the patient in front of them. The frame may be preserved if the
evidence is adequate enough for the clinician to confidently say that the risk data
about the patient matches their existing frame. If there are remaining questions
about whether the patient is at risk, the frame may elaborated as clinicians seek
and infer data in the current context (specific to the individual patient), through
adding and filling slots to build up a more comprehensive frame of the current sit-
uation. These contextual factors are not currently accommodated for by existing
externalisations that come in the form of risk calculator algorithms, guidelines and
population studies.

67. 63
Clinicians apply their internal knowledge, the unstated, that when applied usually
presents itself through skilful performance in order to gain insight. This is what
Schön (1983) refers to as reflection in action. As part of the decision-making pro-
cess, clinicians apply expertise, reflect on their old model, and change their think-
ing to fit the new task. This process is internalised and unique to the individual. In
the same way, Polanyi (1983) speaks of this phenomenon as tacit knowledge.
This leads us to believe that the sensemaking process is more complex than
practitioners are able to verbalise. Henry (2010) supports this finding by saying
appreciation of the tacit dimension of knowledge ‘will help clinicians to build a
more accurate critical framework for evaluating what kinds of information are im-
portant for particular clinical decisions’ (p.296).
When elaborating a frame, clinicians do not simply amass data and elaborate
frames based on best fit, for this would result in erroneous frames being pre-
served. Instead, past experience, clinical expertise, and critical thinking are used
to inform this process. Novices can end up relying on anchors that are not correct
leading distortions or flawed interpretations of what risk data means, leading to
preservation of incorrect frames (Sieck et al., 2007). More experienced clinicians
are able to rely on their larger knowledge base to avoid such pitfalls:
P6 That pattern [some aspect of patient data] can either be caused by only
one thing or it can be caused by 100 things, and that's what comes from
basically learning and doing the job.
Understanding the epistemological aspects of how clinicians think about the data
they use for decision-making and assessment of risk, as well as the environment
and workflow that these processes are embedded in was crucial to the develop-
ment of the tool.

68. 64
Through testing the tool on participants, we have found preliminary evidence that
it supports clinicians by making the implicit (knowledge that they utilise but do not
directly express), explicit. This allowed participants to gain insight and infer rela-
tionships between data through use of the tool.
One participant verbalised the way in which the tool attempts to support the way
in which clinicians apply general rules to specific cases using their internal
knowledge:
P4 I suppose experienced clinicians will have these gestalts, you know. You
see someone and you recognize a certain sort of configuration of features,
and if you then focus on bringing out important risk factors in this patient…
and here's the evidence base behind it... You are facilitating that process.
Risk assessment sits deeply in the context of clinical workflow. This process is
shown in Figure 23. Of course, this is an oversimplification of what actually hap-
pens in clinical practice; however, we found that the general process remained
the same regardless of whether the participant was talking about primary/second-
ary preventative/reactive care. This may have resonated with opinion throughout
partly because it reflects the way in which medicine is taught.
The evidence based medicine process (Gronseth, Woodroffe, & Getchius, 2011)
is aligned with the desired clinical workflow that clinicians expressed. As we can
see, the process is comparable. However, as highlighted by Mynatt (2011), fac-
tors such as diet, activity levels, social conduct other types of intervention, are
generally not viewed in their relation to disease until it presents itself.
In this model, stages may be skipped depending on the availability and confi-
dence that the clinician has in the information they have to work with. For exam-
ple, a clinician may move from Assess Info to Diagnosis if there is strong enough
evidence, or may go from Investigation to Patient monitoring if a test comes back
negative.

69. 65
Figure 23 Model of
desired clinical
workflow expressed
by participants which
aligns with the
evidence based
medicine process
adapted from
(Gronseth,
Woodroffe, &
Getchius, 2011)
The workflow described by participants is as follows:
 Assess info – reviewing existing information establishes a mental model of
the patient, there can be varying levels of data available to the clinician at this
point. This is where the tool developed for this study attempted to target the
needs of clinicians.
 Examination – gathering medical history and taking an account of the pre-
senting complaint through physical examination of the patient. The tool only
emulated this part of workflow where the user had to enter missing blood
pressure information for the patient during the think-aloud.
 Investigation – ordering or performing tests that are not part of the examina-
tion in order to find out something that the current information does not offer.
 Diagnosis – establishing that the patient is suffering from the condition
 Intervention – taking measures to improve decrease risk in a patient.
 Patient monitoring – looking for changes in patient condition. A change may
result in the retrieval of schemas due to a new combination of data elements.
 Patient-clinician communication and consultation – the constant exchange
between patient and clinician that may occur at all times during the clinical
workflow. Patients can be sources of data and collaborate with the clinician in
the data coverage loop ((Russell et al., 1993).
 Clinician-clinician communication – consulting another clinician in order to
gain further information. Communication can help the sense maker become
aware of residue and change schemas (Sharma, 2006). Multidisciplinary
team meetings and second opinions about a patient can lead to reframing.

70. 66
This section contains findings of the way in which clinicians think about risk.
The participants expressed a need for a quantitative measure when they think
about risk. Regardless of whether it was for an overall summary of risk or the risk
related to a single data point that contributes to the risk of a condition, the partici-
pants wanted the numbers. The probabilistic view of risk has the clinician antici-
pating an empirical measure.
One way that clinicians think about risk is absolute risk. This refers to the risk of
being affected by a condition over a period. For example, a 1 in 5, 20%, or 0.2
risk of developing of developing diabetes in the next 10 years.
Relative risk is the ratio of the probability of the patient who is part of a risk group
that is affected by a condition compared to the probability of somebody outside
the risk group being affected. This allows the clinician to put the risk in context
based on the two groups within the population.
The way in which clinicians think about risk is a complex process. The risk of de-
veloping a disease or condition is not the only one that affects the decision-mak-
ing process. Aspects of risk include shared risk factors of related diseases, tem-
poral risk, risk from intervention or medications, the risk of failure to act, risk from
co-morbidities, and economic risk etc. These factors, as well as others, are taken
into account when thinking about risk.
Discussing an exhaustive list of these factors is beyond the scope of this thesis,
so the risk versus benefit of intervention is used as an example. Lifestyle
changes, clinical tests, medication, surgery and other interventions all carry risks
in themselves. When the risk of an intervention gets closer to outweighing the

71. 67
benefits of reducing the risk of developing a condition, clinicians become less
likely to go ahead with that intervention. In primary preventative care, such as in
the scenario used in this study, the tolerance to risk of developing a condition di-
minishes. For example, if the patient has a 1 in 5 chance of developing diabetes
II within 10 years and evidence has shown that an intervention reduces the rela-
tive risk by 50%, the absolute risk goes down from 1 in 5 to 1 in 10. However, a
hypothetical intervention might carry an increased risk of heart attack that out-
weighs the benefits of this risk reduction and therefore the clinician would not tol-
erate the risk of prescribing that intervention. This obviously changes in a situa-
tion where the benefits on intervention are higher than risks of doing nothing. One
participant expanded on this point:
P1 The person who is well, but sort of just in a prevention strategy, we are
not going to give them very major things that carry high risks. Whereas the
individual who I mentioned before whose heart disease condition is so
severe that he's going to be dead in the next three to four months, an
operation with a 25% mortality is balanced by a 100% mortality without it.
Individuals make sense in their own way; sensemaking is inherently an activity
that occurs in the mind of the individual. The way a person makes sense is influ-
enced by the history they have had in medicine, the patients they have seen, and
any other experiences affect a clinician’s understanding of a situation. Therefore,
it should not come as a surprise that participants talked about clinicians having
varying interpretations of risk from the same data:
P3 Different individuals and different clinicians would have different
interpretations of qualitative risks. A small risk or a tiny risk or a moderate
risk... and that's why clinical practice is not consistent between different
doctors, they all have slightly different interpretations of what's going on
and the risks of different things.

72. 68
P6 What I often find is that different clinicians seem to remember risks a bit
differently. Sometimes one will say ‘oh, this [a risk factor] is more important’
and the other would say ‘this [the same risk factor] is less important’.
This inconsistency between clinicians is due in part to the nature of clinical prac-
tice, which is a subjective activity, despite the aspiration to remain objective when
assessing risk. Our findings echo Sutherland & Dawson (2002) who state, ‘in the
doctors’ worlds, new information is received and interpreted on the basis of past
experiences, cognitive structures, and social context’.
In some cases the threat of accountability, may create a lower tolerance toward
risk. It is possible for lawsuits to occur due to ‘greed, or simply because the pa-
tient or family members did not like the outcome or the doctors involved’ (Noland
& Carl, 2006, p.88). As a participant said, underestimating risk can result in nega-
tive consequences for the clinician, but overestimating may cause negative con-
sequences to the patients:
P1 I think for medical legal reasons, the world gets a little more defensive
and therefore we don't want to underestimate risk of a treatment procedure
and risk the patient or colleagues saying 'you only quoted 10%, it was
clearly 25%'. […] Maybe then there's a worry that we overestimate the risks
to be defensive. Much better to have been conservative than provide
someone with a higher risk, but if you start doing that [i.e. overestimating]
we might get more people saying no and declining important care.
Findings show that the participants deal with an excess of data from multiple of
sources. Extracting the most important information from seemingly irrelevant ma-
terial remains a challenge. The presentation of data, availability of information,
and trust placed in information all have severe implications on clinicians’ deci-
sion-making.

73. 69
Data from interviews and think-aloud sessions revealed that trust is assigned to
data in different ways. The weighting attached to data in the decision-making pro-
cess surrounding risk is connected to how much they trust it.
Trust in self-reported data from patients tends to be lower than other types of
data because of the expectation of a higher margin of mistake in them: patients
may lie for their own reason, misreport, forget, or lack the appropriate knowledge.
P1 People lie and in consultations, not everyone tells the truth the whole
time, but then it's possible that you get an incorrect value based on an
incorrect piece of information […] You get a lot of people that say 'ah,
everything's fine' and then you scratch below the surface and it’s not. Trust
is a judgement, but anything that involves a person […] what I'm saying
there is a higher risk versus sending a blood sample to a laboratory and
getting a number back.
There is also a chance of misinterpretation during communication between pa-
tient and clinician. This happens when the mental model of the patient does not
match that of the clinician. Coping strategies to mitigate these issues include ask-
ing the patient in a number of different ways and approaching witnesses (e.g.
asking friends, family and co-workers) to confirm what the patient reports. In our
study, clinicians regarded expertise as the key factor in being able to know how
much weight to place on patient reported data.
The quality of the tools used to gather data is another factor clinician’s take into
account when assigning trust. For example, a participant was describing a patient
using their personal sphygmomanometer to measure their own blood pressure
(BP) from home. The sphygmomanometer is not quality assessed like the ones in
the hospital; therefore, trust in the patient’s equipment is lower. With the increase
of consumer eHealth devices, trust in the quality measurement tools will likely be-
come a prevailing issue. One possible way to mitigate the uncertainty created by
this lack of trust is by triangulating data from a number of sources.

74. 70
Clinicians do not only assign trust to the tool itself, but also the way in which it is
used. The quality of the tool may be fine, but the measurement method may be
flawed:
P5 The issue that some patients take the BP at the wrist as opposed to
taking it on their arm. The risk reading is going to be a different reading to
the arm. The data that the patients sometimes present might not be as
accurate as the data that we have.
Other ways that clinicians assign trust are:
 Volume of data – more usually means more confidence as trends over time
are revealed. For example, P5 said that BP readings from a patient might be
preferable if there are many data points making anomalies easier to track.
 Age of data – depending on the type of data, older data can be less trusted
than newer data, for example blood chemistry tests from years ago have little
bearing on the patients current condition.
The findings suggest that clinicians say that they assign trust to data is partly due
to the amount of uncertainty surrounding a data point. Since the limitations are
known for clinically generated quantitative data, clinicians assign higher trust:
P1 [the highest trust is assigned to] hospital generated data, which is hard
data. Results from the laboratory, results from a scan, results from a
procedure where we will review that, we will know […] some of the
limitations of that.
Although there was a consensus that all data should be viewed with scepticism, a
number of participants spoke of quantifiable hard data as the basis of clinical
knowledge, and therefore overall more trustworthy. At the opposite spectrum, we
find ‘soft’ qualitative data. The tendency (due in part to medical training) is to as-
sign to numerical, clinical and quantified data (i.e. ‘objective’ data) a higher
amount of trust than qualitative, textual and contextual ones (i.e. patient reported
and ‘subjective’ data).

75. 71
However, as observed through the study via interviews and in the think-aloud
session, the way that clinicians claim they assign trust to and use data to reach a
decision does not fully draw a parallel to the way in which this is applied in prac-
tice. Much of the time, the soft data is invaluable for gaining insight or recognising
further information needs. Clinicians showed varying levels of acceptance to the
idea of formalising this soft data as part of clinical practice, because they viewed
the data through an evidence-based medicine lens.
Some participants welcomed the idea of a tool that supported the process of
making the implicit, explicit. In contrast, others resisted the idea of including this
data in an interface. There seems to be a disconnect between the way that some
clinicians verbally and rationally assign importance to hard data versus soft data.
Some said the only data they need to see is ‘validated’ ‘hard’ data, but when
asked to think of which data they use in practice; it included ‘soft’ data:
P5 In terms of the patient information, knowing that she is widowed is fairly
important. Her husband might have just died last week, and if her husband
died last week you don't really want to necessarily be talking to her about
her risk of melanoma, she won't care…
We are familiar with the problem of noisy, multivariate, homogeneous data gener-
ated from dissimilar sources. It is good to have all information available, but pre-
senting it in the wrong way can easily increase cognitive load. As we know, risk is
not only the sum total of all of the available data, some data points will trump oth-
ers in terms of urgency or severity.
Things get even more complex when we start to think about types of data that
most clinicians are not used to looking at. We found that the majority of clinicians
have limited understanding of genetic risk except for monogenic, or single abnor-
mal gene, diseases (i.e. Mendelian disorders), unless they were specialists in the
field. Participants expressed a desire for supporting explanation and summarisa-
tion of genetic risk, although participants who were not genetics experts lacked
the confidence to interpret genetic information.

76. 72
Rosas-Blum, Shirsat, & Leiner (2007) observe that there is a lack of essential
knowledge and comprehension of genetics; this was reflected in our findings:
P4 The more you understand these things the more you accept them. They
were talking about doing genome sequencing on the radio this morning, I
don't want to know to be honest. I'm a head in the sand person. Unless of
course it's something very very preventable.
Genetic risk is still in the early stages and also susceptible to the same problems
of trust, in terms of the strength of the study and supporting evidences, rele-
vance, and of the patient being outside the population of the study being used as
reference:
P2 Most data about genetics [specifically referring to congenital heart
defects] are very basic research data on genes and gene pathways, which
may or may not have anything to do with the patient you're looking at, or
they're based on large population averages, so they're a gross estimate of
risk.
In order to move from assessing risk to making a decision, a clinician must have
enough information and enough confidence in it. This means clinicians work in
conditions of great uncertainty and within a probabilistic paradigm (e.g. establish-
ing relative plausibility) (Hammersley, 1995).
Encountered information that brings the possibility of a patient being at risk to
their attention usually creates a latent need for more information and thus, more
information seeking. Sometimes the clinician is not aware of available information
and so the need does not arise. If it does arise later, it can be a source of great
frustration.

77. 73
One participant said that it is common to do many investigations and then find out
some unrelated contextual data:
P6 I would say the biggest problem with information not being useful is
when information is left out […] Often I find when I'm reviewing notes, what
is missing, is the most frustrating “Oh, why did they not ask about this”,
“Why did they not check this?” Purely because people forget when they are
in a rush.
Frames can be elaborated when new data that was missing before is presented,
leading to new inferences and relationships being made. In this example, the
small biopsy and uncertainty of diagnosis stimulates thinking about missing data.
We explain the following process in the context of Pirolli & Card's (2005) sense-
making model; information, scheme, insight, product:
P6 …with this very tiny biopsy that was taken […] we could not make a
diagnosis so we asked the surgeon whether he could go back and take a
larger chunk of tissue […] when that tissue came through we took a frozen
section and within five minutes we had a definitive diagnosis. […] While the
diagnosis of infantile fibrous sarcoma is by and large very easy it can still be
impossible if the tissue is not representative or insufficient, however you
don't know what you're diagnosing in advance so you have that source of
uncertainty.
The clinician did not have enough data to make sense of the situation, and there-
fore asked the surgeon for a bigger sample because of an information need. The
new data was re-represented in the appropriate schema, through manipulation of
the representations of about what the patient’s tumour actually was, insight was
gained. Following this newfound understanding, a product was created in the
form of a definitive diagnosis.
Although this example seems very straight forward, it is impossible to know if
substantial data is missing unless the relevant frame is elicited through the appro-
priate anchors. There is a constant strive to ‘completeness’ and a desire to be
aware of what relevant information is not present, and to seek it out.

78. 74
Tools can assist in flagging missing information used in validated scales and
other models. However, it is not often known what other data are needed as clini-
cians use so many contextual factors as anchors in so many different combina-
tions that it is probably unrealistic to compile an exhaustive list in any externalisa-
tion. It seems that there is a misapprehension (or at least an unachievable desire)
that data about a patient can be all encompassing. Usually, when a gap in
knowledge is identified, a way to fill that gap needs to be present. Filling a gap in
knowledge will then often lead to further information needs. The sensemaking
process is cyclical by nature.
Visualisation was used in the study to display, interact with and explore infor-
mation in a graphical manner with the hope that it would aid the participants’ un-
derstanding of risk. By mapping data to a visual form, the tool attempted to sup-
port visual sensemaking. By making sense of externalisations, internal models of
the domain are formed (Ware, 2013), preliminary evidence shows that the tool
supports this.
 The tool presented patient information (information that is usually gathered
and stored in the clinicians’ mental model of the patient) whilst steering clini-
cians’ attention to risk factors (risk factors for a particular condition with an ev-
idence base as to why it is a risk factor) that were relevant to their assess-
ment of risk through an appropriate information structure. The way that partic-
ipants interacted with the tool showed that presenting both validated/clinical
scales/data and contextual information as part of the externalisation was rele-
vant to their needs.
 It also provided the relevant evidence to support why a data point had risk at-
tributed to it, along with access to guidelines and standards. This allowed the
clinician to gauge the data quality.
 The data that was and was not taken into account by quantitative risk scales
(risk calculator) was plainly highlighted, therefore showing how comprehen-
sive that empirical measure was.

79. 75
Figure 24 Seeking a frame.
The condition identified by the
tool and the available patient
data act as anchors.
The goal of the clinician, when assessing risk, is to make sense of the information
they have about an individual patient and then decide on the next steps to take in
order to modify the patient’s life. Here we will present findings about the tool de-
veloped for this thesis. This section will cover the things that worked well to sup-
port making sense of risk, and cover some of the limitations and shortcomings of
the tool.
The overview page was found to provide a well-
structured representation of information to help
the clinician in thinking about risk.
P4 Linear clearly presented demographic
versus risk factor information, so that's fairly
clear.
P5 I like the fact that its patient information
and risk factors, it's absolutely clear.
Presenting all of the patient information in the
same interface helped to provide the contextual
factors that clinicians use to think about risk in
their work, but are not taken into account by tra-
ditional tools. Provide overview (Yi et al., 2008)
helped clinicians see what parts were most rele-
vant for further inspection by providing a ‘con-
cept-oriented and task-specific’ summary that
may help clinicians ‘recognize new problems, im-
plement preventive care and formulate care
plans’ (Feblowitz et al., 2011).
Providing the patient information (and risk factors) along with the risk condition
served as anchors to stimulate thinking about risk, thus connecting the data to a
frame about patient risk of developing a condition (Figure 24).

80. 76
Figure 25 Data
categories.
The patient information was organised into categories that followed the pattern
that clinicians are used to seeing it, making it easier for users in the sensemaking
loop to search and filter information. This was part of supporting the clinical work-
flow, generally moving from broad to specific (Figure 25). The designs for the vis-
ualisation attempted to decrease the gap between users mental model – match
mental model (Yi et al., 2008) – and data in order to decrease cognitive load.
Essential to the task, risk was represented by the visualisation in a frame on the
right hand side with most significant information that clinicians use when thinking
about risk given priority. Participants expressed that the most severe risk factors
are the most essential to take into account because they will lead to insight about
the patient’s potential future condition:
P1 If the risk is low; you wouldn't be delving into why it's low. Whereas if
it's elevated then you'd start to look into what you can do to modify that.
Of these high risk factors, the changeable ones were the most important, since
they are the ones that would allow for intervention:
P1 You could argue that, the most important ones are the changeable ones
because there's something you can do about them.
Non-changeable risk factors are also important for educating patients about their
risk, even if it is not possible to mitigate that risk through intervention:
P4 One of the worst things is in fact if your chance of 1 in 10 is all to do
with your male, ethnicity and your genes. Then I suppose what would you
do? But, it's worth knowing that

81. 77
The we found that the tool supported this information need.
Figure 26 shows elaborating a frame by satisfying the queries
that arose when questioning the frame about a patient being at
risk of developing a condition. This is how the tool design tack-
led the need:
Risk severity – The ordinal scale of high/medium/low risk was
used to represent severity, clinicians deemed high risk factors
most important and low risk factors least important. A colour
mapping of differing saturation was used to let the clinician
know which risk factors were more severe.
Changeable risk factors – important since they were also the
factors that were directly actionable through intervention. Vis-
ual differentiation was achieved by grouping of the modifiable
factors.
Non-changeable risk factors – secondary in importance as
they are useful for educating the patient about their risk, even
if it is not possible to change them. These were grouped in the
same way at changeable factors, but contrasted from the
changeable through placement and a separating line.
Most western audiences natively read left to right and top to bottom. With hind-
sight, the structure of information may have been better if changeable factors had
been positioned above non-changeable factors, and within the two respective
sections, high/medium/low risk factors ordered from top to bottom, thus, giving
visual priority to the most important information to the western user.
Figure 26 Elaborating
a frame.

82. 78
Providing a risk summary was found to be a need for clinicians. A risk number is
useful for giving an empirical measure of the information. Some participants
noted that some clinicians rely on calculators a lot:
P1 There will be some clinicians who go 'I don't care about that [points to
patient information], I care about that [points to risk calculator score]'.
As mentioned earlier, calculators, evidence and even guidelines were developed
with a certain population, not all patients fit within these populations:
P4 I suppose sometimes it would depend on what population you're dealing
with. Sometimes ranges depend on the population that you're dealing with.
If we were to take a range of haemoglobin in blood, our range in western
society is a particular range. But you translate that range to an Indian
society, it would be completely different. Because we live in a multicultural
society sometimes, you can't necessarily believe the ranges that are
applicable to this population. Sometimes you have to be sensitive to that.
We look at many western calculators of risk, which is not necessarily always
appropriate to minorities.
Here we highlight an example of how data-frame theory
forces a change in the clinicans understanding of the world.
In light of the new information about the patient not fitting into
the risk population that the evidence is based on,
the clinian finds that the situation is not what
was anticipated in the original frame. Once
the clinian realises that the patient
does have the risk that the evidence
would suggest, the clinician’s world
view must be adjusted; this is done
through reframing. In Figure 27 we
can see that when anomalies or
Figure 27 Reframing
cycle in progress.

83. 79
inconsistencies occur, the competing data forces a change in understanding.
Before the understanding was that the patient was at risk, due to the evidence
presented in the original anchors. However, the frame no longer fits the new data
no longer fits; the patient is outside the population for the evidence base and
therefore, reframing occurs.
Participants said that using other patient information they are able to explain and
communicate to the patient why the risk score given by the calculator may not
apply to them:
P5 You're just giving them an idea of their risk... you adapt your risk
calculator. You have the conversations which says 'this might be the case'...
you put in a caveat. It would be good to have a bit of narrative around that,
in the risk calculator say. This is a risk calculator, it's from this particular
population and the narrative might be that if you’re thinking about a
different population, you might just need to just take that into account. You
might not be able to completely calculate that, but it's a narrative, it's
qualitative.
In addition, participants noted that risk calculators vary in quality:
P4 There are lots of different risk calculators, some of them are more
evidence based than others.
The tool successfully showed where the risk calculator algorithm was lacking by
displaying factors that research studies have identified as risk factors but were
not taken into account by risk calculators thus, highlighting the limitations of the
summary and laying emphasis on the need for further clinical judgement in the
areas not covered by the risk calculator. When running the calculator the user
was prompted to fill in information. This helped to make participants aware of
missing data.
Zooming into the individual data point view allowed the participants to explore
why something contributes to risk. This operation had the intent of enabling the

84. 80
user to abstract/elaborate one aspect of the patient information to show more de-
tail by interacting with the interface. The user was also able to explore the other
patient information from this view, supporting the information foraging loop. Infor-
mation for individual data points was embedded within visualisations, these
aimed to impart knowledge about the distribution of risk for that factor within a
population, as well as where the patient sits within the population.
P4 I like the fact that you've got, two things, you've got the knowledge bit,
the overall prevalence. Then you've identified which group the patient is in.
Think-aloud data revealed that the inclusion of supporting sources within the tool
was perceived as useful to all participants. However, some participants men-
tioned that certain clinicians might like to use guidelines and find viewing evi-
dence is too much work due to the sheer volume of available evidence and time it
takes to go through it. One participant mentioned that it is not common to view
evidence during a consultation. Whereas others would like to go through evi-
dence to aid their decision-making process. Further improvement of evidence
source and clinical trial research visualisation for sensemaking could aid clini-
cians in searching for evidence to support risk assessment. An approach similar
to that of Faisal, Cairns, & Blandford (2006), but for the healthcare research evi-
dence domain may be one possible way to approach this problem.
In summary, the tool supported the clinicians in making sense of risk. All availa-
ble patient information was make available, without drawing the user attention
from the task. Information was presented in a structured manner by grouping re-
lated items and contrasting dissimilar entities such as changeable/non-changea-
ble and data contributing to risk/data not contributing to risk. Segmentation of
lists, blank space and lines were used to avoid overloading the screen with con-
tent. The tool was integrated into the clinical workflow by providing an appropriate
order of screens, information and alerts. Missing information was highlighted by
prompting the clinician to enter information that was needed for the risk calculator

85. 81
at a suitable moment. However there was also issues with the tool, such as use
of colour and consistency of terminology and wording, most of which could most
likely be fixed through adding further iterations to the design process. Further
consultation on the correct clinical terminology throughout the tool would have
avoided the need for the clinicians to switch regimes of language during use of
the tool.

86. 82

87. 83
6 LIMITATIONS
In this chapter, we first cover some of the design limitations of the tool. Then
weaknesses of the study as a whole as discussed, and then we propose ways in
which the tool could be improved to further meet the needs of clinicians.
There were a number of cases where the tool failed to support the clinicians’ task
of assessing risk. There were also instances where the system model failed to
match that of the user. This section will look at some of the shortcomings of the
tool.
Results show that the option of manually changing a risk of a data point in the
tool was a controversial point. Some participants were opposed to the implica-
tions of defying a validated data point, even if it was flagged for review in the sys-
tem. It was preferable for most participants to change the value of a data point ra-
ther than the risk score attached to it:
P3 If the data's correct you should use them as they are […] You're not
going to change the classification of the weight, you want to change the
weight to make it correct. I'm not sure how useful this is unless it's very
inherent as part of what the actual data item is.
This feature is obviously not useful for things such as weight, but maybe there is
potential for use with more contextual such as stress at work:
P3 Stress at work could be a risk factor for other things, so... It's a question
of 'how is that best represented?' You could have 'stress at work' and assess
that... the actual job doesn't matter […] then again job is demographic
information. It would be relevant for lots of other kind of clinical uses apart
from risk assessment. You can't only go by the job description, because two
jobs could have the same description but could have a very different work

88. 84
environment. I'm not sure about this 'changing a risk classification... This is
a case where it probably does make sense, but this kind of thing wouldn't
work for other things. Like medical conditions.
Some participants thought it was best to ‘override the system’ not through a tool,
but by talking with the patient directly and having a conversation about why the
risk is different for them and the risk the evidence says that they have does not
apply to them. However, some participants felt that it was essential to allow the
clinician to flag items for review, stating that feedback is essential, both to the de-
velopers/maintainers of the tool, as well as to fellow clinicians for second
opinions:
P4 I think the more you can collect feedback from people... Feedback about
not understanding what the system is doing […] Just ongoing feedback
about whether it's producing [useful results]. Whether it is actually doing
something useful basically.
Using colour to represent risk was found to be rather haphazard, some partici-
pants understood the use of colour, and some did not. A few participants did not
understand that the three gradations of colour represented low, medium and high
risk. The fine distinctions in increasing saturation levels was not directly mapped
to increased risk in all participants’ mental model. Although this method was more
effective than using size (Stage 2, Visualisation 2), it was less effective than us-
ing placement (Stage 2, Visualisation 1) as a signifier for low, medium and high
risk. Using a key to make the differences in saturation obvious may be an option,
but further visual design for effective use of colour (Silva et al., 2011) is required
for an optimal solution.
Using the red and green in the same interface to ascribe meaning to risk severity
was a problem. One participant was colour-blind, resulting in the risk severity fail-
ing to hold meaning for them. However, most participants understood red as be-
ing risky and green as being safe.

89. 85
Showing risk in colours other than red and green may not have been semantically
resonant (Lin, Fortuna, Kulkarni, Stone, & Heer, 2013) to clinicians looking at risk.
Some participants even called for the use of those colours when exploring the in-
terface:
P8 Make the others [the reduced risk factors in the risk calculator] green
and the risk factors red and that would be more obvious.
Clinicians have a common way of talking about risk. Being educated in the same
manner has led to a shared language; the language of science is specific. Partici-
pants mentioned a similar phenomenon of regime switching where patients have
different terms for the same concept. Similarly, the differences in nomenclature
were the cause behind some of the misunderstandings in the interface that may
have been avoided with better choice of language:
P4 Changeable and non-changeable, those are static and dynamic really
aren't they? Which are terms I'm more familiar with.
A limitation of the recruitment method used in the study is that the clinicians
who were recruited through industry connections were inherently interested in
technology in healthcare. This may have led to a bias in prevailing attitudes
toward the tool.
Although we found that the general principles of assessing risk using complex
health data is relevant to both primary and secondary care, the tool used in this
thesis focused on a primary preventative setting. However, much of the way
healthcare is delivered today is reactive rather than preventative. This may have
influenced the way in which participants thought about the tool. That said, we
believe the tool used is both relevant and timely since primary preventative
healthcare is high on the agenda, both in the United Kingdom (Department
of Health, 2009) and the United States of America (Francis Collins, 2014).

90. 86
Since the personas were created for the purposes of this study, the data set is
not based on real patient data. Although a doctor and a nurse reviewed the per-
sonas for validity, the use of fabricated data may have influenced the way that
participants thought about and understood the tool. Testing of the tool was limited
to using scenarios with personas and a prototype tool. Testing using a real pa-
tient in a clinical setting would have been preferable put was not feasible due to
pragmatic reasons such as temporal constraints and ethical concerns. These lim-
itations in the experimental design have implications on the transferability of find-
ings to real clinical practice. Clinical trials would be needed to validate the useful-
ness of this tool in an environment with actual patients and working clinicians.
Ideally the tool would not only be able to support clinicians in deciding if a patient
is at risk, but also to help determine which prevention measures would provide
the greatest reduction in risk. Integration of the tool into the entire clinical work-
flow would have been ideal, something that was beyond the scope of this study.
Integrating existing guidelines such as those proposed by Horsky et al. (2012) to
fit existing standards would be useful for this process.
Workplace factors also played a part in the perceived utility of the tool; partici-
pants mentioned that they do not have access to tablets or computers during
their consulting work. The ability to output, or what Shneiderman (1996) calls ex-
tract, the relevant sub-collection of data would be necessary to integrate the tool
into a real work environment. The data showed that, this tool would only be useful
if it pulled data from existing sources and if clinicians could input data back to the
Electronic Heath Record. Although these issues are important for the successful
integration of tools into the healthcare system (such as the one proposed in this
study), there are policy, data ownership, security, and authorisation considera-
tions that need to be taken into account. These topics are multi-layered and com-
plex and therefore far beyond the scope of this study. The study design did not
take into account the important role that society and organisations in healthcare
play when it comes to clinicians dealing with complex data and thinking
about risk.

91. 87
7 CONCLUSIONS
By looking through the sensemaking lens, we have provided an account of the
way in which clinicians make sense of complex data when assessing risk in a pa-
tient. We have shown that assessing risk is not merely about knowing the rules,
but about deciding which rule is most relevant, or when the rules do not apply.
When a representative sample of a population is available, clinicians are able to
understand the implications of risk for the abstract patient. However often the sit-
uation is more nuanced, the average for the population might not fully apply to
the patient in front of them. By using an array of information – derived from both,
what clinicians consider hard and soft data – they consciously or subconsciously
determine which patient characteristics influence their decisions.
It was found that some clinicians tend to downplay or be unaware of factoring
‘soft’ or non-clinical data into their judgments. The processes behind these judg-
ments are tough to express explicitly but are important for personalised, patient-
centred work. Through this process clinicians are able to interpret when general-
ising the abstract to the individual patient (Smith & Egger, 1998), what is and is
not appropriate.
This thesis presented the design process taken to develop a visualisation to sup-
port the clinicians’ needs was presented. We have shown preliminary evidence
for a tool that supports the sensemaking process, by reducing the gap between
the representation of data and the clinicians’ mental model. However, further in-
vestigation is required to assess the utility and safety of such a tool in actual
clinical settings.

92. 88

93. 89
8 BIBLIOGRAPHY
Anders, K., & Simon, H. A. (1980). Verbal reports as data. Psychological Review,
87(3), 215–251. doi:10.1037/0033-295X.87.3.215
Bertin, J. (2010). Semiology of graphics: diagrams, networks, maps (1st ed.).
Redlands, Calif: ESRI Press : Distributed by Ingram Publisher Services.
Boren, T., & Ramey, J. (2000). Thinking aloud: Reconciling theory and practice.
Professional Communication, IEEE Transactions on, 43(3), 261–278.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative
Research in Psychology, 3(2), 77–101. doi:10.1191/1478088706qp063oa
Bui, A., & Hsu, W. (2010). Medical Data Visualization: Toward Integrated Clinical
Workstations. In Medical Imaging Informatics (pp. 139–193). New York:
Springer US.
Card, S. K., Mackinlay, J. D., & Shneiderman, B. (1999). Readings in information
visualization: using vision to think. San Francisco, Calif: Morgan Kauf-
mann Publishers.
Department of Health. (2009). NHS 2010-2015: from good to great: preventative,
people-centred, productive. London: Stationery Office.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: verbal reports as data
(Rev. ed.). Cambridge, Mass: MIT Press.
Faiola, A., & Hillier, S. (2006). Multivariate relational visualization of complex clini-
cal datasets in a critical care setting: A data visualization interactive proto-
type (pp. 460–468). IEEE. Retrieved from http://ieeex-
plore.ieee.org/xpls/abs_all.jsp?arnumber=1648300

94. 90
Faisal, S., Blandford, A., & Potts, H. W. (2013). Making sense of personal health
information: Challenges for information visualization. Health Informatics
Journal, 19(3), 198–217. doi:10.1177/1460458212465213
Faisal, S., Cairns, P., & Blandford, A. (2006). Developing User Requirements for
Visualizations of Literature Knowledge Domains. In Information Visualiza-
tion, 2006. IV 2006. Tenth International Conference on (pp. 264–269).
IEEE. Retrieved from http://ieeexplore.ieee.org/xpls/abs_all.jsp?ar-
number=1648271
Faisal, S., Cairns, P., & Blandford, A. (2007). Challenges of evaluating the infor-
mation visualisation experience. In Proceedings of the 21st British HCI
Group Annual Conference on People and Computers: HCI... but not as
we know it-Volume 2 (pp. 167–170). British Computer Society. Retrieved
from http://dl.acm.org/citation.cfm?id=1531451
Feblowitz, J. C., Wright, A., Singh, H., Samal, L., & Sittig, D. F. (2011). Summari-
zation of clinical information: A conceptual model. Journal of Biomedical
Informatics, 44(4), 688–699. doi:10.1016/j.jbi.2011.03.008
Francis Collins. Fiscal Year 2015 Budget Request, § Senate Subcommittee on
Labor – HHS – Education Appropriations (2014). Maryland. Retrieved
from http://www.nih.gov/about/director/budgetrequest/fy2015testi-
mony.htm
Fry, B. J. (2004). Computational information design. Massachusetts Institute of
Technology. Retrieved from http://dspace.mit.edu/handle/1721.1/26913
Gillham, B. (2005). Research interviewing: the range of techniques. Maidenhead ;
New York: Open University Press.

95. 91
Gronseth, G. S., Woodroffe, L. M., & Getchius, T. S. (2011). Clinical Practice
Guideline Process Manual. St. Paul, MN: American Academy of Neurol-
ogy. Retrieved from http://curebraindisease.net/globals/axon/as-
sets/9023.pdf
Guion, L. A., Diehl, D. C., & McDonald, D. (2011). Triangulation: Establishing the
validity of qualitative studies. Retrieved from http://edis.ifas.ufl.edu/fy394
Hammersley, M. (1995). Ethnography: principles in practice (2nd ed.). London ;
New York: Routledge.
Harle, C. A., Neill, D. B., & Padman, R. (2008). An information visualization ap-
proach to classification and assessment of diabetes risk in primary care.
In 3rd INFORMS Workshop on Data Mining and Health Informatics (DM-
HI 2008) (pp. 1–6).
Harle, C. A., Neill, D. B., & Padman, R. (2012). Information Visualization for
Chronic Disease Risk Assessment. Intelligent Systems, IEEE, 27(6), 81–
85.
Hartson, H. R., & Pyla, P. S. (2012). The UX Book: process and guidelines for
ensuring a quality user experience. Amsterdam ; Boston: Elsevier.
Henry, S. G. (2010). Polanyi’s tacit knowing and the relevance of epistemology to
clinical medicine: Tacit knowing and clinical medicine. Journal of Evalua-
tion in Clinical Practice, 16(2), 292–297. doi:10.1111/j.1365-
2753.2010.01387.x
Horsky, J., Schiff, G. D., Johnston, D., Mercincavage, L., Bell, D., & Middleton, B.
(2012). Interface design principles for usable decision support: A targeted

96. 92
review of best practices for clinical prescribing interventions. Journal of Bi-
omedical Informatics, 45(6), 1202–1216. doi:10.1016/j.jbi.2012.09.002
Huang, W., & Eades, P. (2013). Handbook of human centric visualization. Re-
trieved from http://dx.doi.org/10.1007/978-1-4614-7485-2
Johnson-Laird, P. N. (1983). Mental models: towards a cognitive science of lan-
guage, inference and consciousness. Cambridge [Cambridgeshire] ; New
York: Cambridge University Press.
Jupp, V. (2006). The Sage dictionary of social research methods. London ; Thou-
sand Oaks, Calif: SAGE Publications.
Kirsh, D. (2010). Thinking with external representations. AI & SOCIETY, 25(4),
441–454. doi:10.1007/s00146-010-0272-8
Klein, G., Moon, B. M., & Hoffman, R. R. (2006). Making Sense of Sensemaking
1: Alternative Perspectives. IEEE Intelligent Systems, 21(4), 70–73.
Klein, G., Phillips, J. K., Rall, E. L., & Peluso, D. A. (2007). A data-frame theory of
sensemaking. In R. R. Hoffman (Ed.), Expertise out of context: proceed-
ings of the sixth International Conference on Naturalistic Decision Making
(pp. 113–155). New York: Lawrence Erlbaum Associates.
Krygier, J., & Wood, D. (2005). Making maps: a visual guide to map design for
GIS. New York: Guilford Press.
Lin, S., Fortuna, J., Kulkarni, C., Stone, M., & Heer, J. (2013). Selecting Semanti-
cally-Resonant Colors for Data Visualization (Vol. 32, pp. 401–410). Wiley
Online Library. Retrieved from http://onlineli-
brary.wiley.com/doi/10.1111/cgf.12127/full

97. 93
MacDonald, L. W. (1999). Using color effectively in computer graphics. Computer
Graphics and Applications, IEEE, 19(4), 20–35.
Mazza, R. (2004). Introduction to Information Visualisation. Faculty of Communi-
cation Sciences, University of Lugano. Retrieved from
http://www.dti.supsi.ch/~mazza/infovis_introduction.pdf
Mynatt, E. D. (2011). IT in healthcare: a body of work. Interactions, 18(3), 22–25.
National Health Service. (2013, August 4). Type 2 diabetes self-assessment. Re-
trieved from http://www.nhs.uk/Tools/Pages/Diabetes.aspx
Nielsen, J. (1993). Iterative user-interface design. Computer, 26(11), 32–41.
Nielsen, J. (1994). Guerrilla HCI: Using discount usability engineering to pene-
trate the intimidation barrier. Cost-Justifying Usability, 245–272.
Nielsen, J., & Faber, J. M. (1996). Improving system usability through parallel de-
sign. Computer, 29(2), 29–35.
Noland, C., & Carl, W. J. (2006). “It’s Not Our Ass”: Medical Resident Sense-
Making Regarding Lawsuits. Health Communication, 20(1), 81–89.
doi:10.1207/s15327027hc2001_8
Pirolli, P., & Card, S. (2005). The sensemaking process and leverage points for
analyst technology as identified through cognitive task analysis. In Pro-
ceedings of International Conference on Intelligence Analysis (Vol. 5, pp.
2–4). Mitre McLean, VA. Retrieved from http://vadl.cc.gatech.edu/docu-
ments/2__card-sensemaking.pdf
Polanyi, M. (1983). The tacit dimension. Gloucester, Mass.: Peter Smith.
Riessman, C. K. (1993). Narrative analysis. Newbury Park, CA: Sage Publica-
tions.

98. 94
Rind, A., Wang, T., Aigner, W., Miksch, S., Wongsuphasawat, K., Plaisant, C., &
Shneiderman, B. (2013). Interactive Information Visualization to Explore
and Query Electronic Health Records. Foundations and Trends® in Hu-
man–Computer Interaction, 5(3), 207–298. doi:10.1561/1100000039
Rosas-Blum, E., Shirsat, P., & Leiner, M. (2007). Communicating genetic infor-
mation: a difficult challenge for future pediatricians. BMC Medical Educa-
tion, 7(1), 17. doi:10.1186/1472-6920-7-17
Russell, D. M., Stefik, M. J., Pirolli, P., & Card, S. K. (1993). The cost structure of
sensemaking. In Proceedings of the INTERACT’93 and CHI’93 confer-
ence on Human factors in computing systems (pp. 269–276). ACM. Re-
trieved from http://dl.acm.org/citation.cfm?id=169209
Saraiya, P., North, C., Lam, V., & Duca, K. A. (2006). An insight-based longitudi-
nal study of visual analytics. Visualization and Computer Graphics, IEEE
Transactions on, 12(6), 1511–1522.
Schön, D. A. (1983). The reflective practitioner: how professionals think in action.
New York: Basic Books.
Sharma, N. (2006). Sensemaking: Bringing theories and tools together. Proceed-
ings of the American Society for Information Science and Technology,
43(1), 1–8.
Shneiderman, B. (1996). The eyes have it: A task by data type taxonomy for in-
formation visualizations. In Visual Languages, 1996. Proceedings., IEEE
Symposium on (pp. 336–343). IEEE. Retrieved from http://ieeex-
plore.ieee.org/xpls/abs_all.jsp?arnumber=545307

99. 95
Shneiderman, B., Plaisant, C., & Hesse, B. (2013). Improving health and
healthcare with interactive visualization methods. IEEE Computer Special
Issue on Challenges in Information Visualization, 58–66.
Sieck, W. R., Klein, G., Peluso, D. A., Smith, J. L., Harris-Thompson, D., & Gade,
P. A. (2007). FOCUS: A model of sensemaking. DTIC Document. Re-
trieved from http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPre-
fix=html&identifier=ADA469770
Silva, S., Sousa Santos, B., & Madeira, J. (2011). Using color in visualization: A
survey. Computers & Graphics, 35(2), 320–333.
doi:10.1016/j.cag.2010.11.015
Smith, G. D., & Egger, M. (1998). Incommunicable knowledge? Interpreting and
applying the results of clinical trials and meta-analyses. Journal of Clinical
Epidemiology, 51(4), 289–295.
Spence, R. (2007). Information visualization: design for interaction (2nd ed.). New
York: Addison Wesley.
Spence, R. (2014). Information visualization: an introduction. [Place of publication
not identified]: Springer.
Straus, S. E. (2011). Evidence-based medicine: how to practice and teach it. Ed-
inburgh: Elsevier Churchill Livingstone.
Sutherland, K., & Dawson, S. (2002). Making sense in practice: Doctors at work.
International Studies of Management & Organization, 51–69.
Treisman, A. (1998). Feature binding, attention and object perception. Philosophi-
cal Transactions of the Royal Society of London. Series B: Biological Sci-
ences, 353(1373), 1295–1306.

100. 96
Tufte, E. R. (1995). Envisioning information. Cheshire, Conn.: Graphics Press.
Wagemans, J., Elder, J. H., Kubovy, M., Palmer, S. E., Peterson, M. A., Singh,
M., & von der Heydt, R. (2012). A century of Gestalt psychology in visual
perception: I. Perceptual grouping and figure–ground organization. Psy-
chological Bulletin, 138(6), 1172–1217. doi:10.1037/a0029333
Ware, C. (2013). Information visualization: perception for design (Third edition.).
Waltham, MA: Morgan Kaufmann.
Weick, K. E. (1996). Sensemaking in Organizations. The Academy of Manage-
ment Review, 21(4), 1226. doi:10.2307/259169
Yi, J. S., ah Kang, Y., Stasko, J. T., & Jacko, J. A. (2007). Toward a deeper un-
derstanding of the role of interaction in information visualization. Visuali-
zation and Computer Graphics, IEEE Transactions on, 13(6), 1224–1231.
Yi, J. S., Kang, Y., Stasko, J. T., & Jacko, J. A. (2008). Understanding and char-
acterizing insights: how do people gain insights using information visuali-
zation? (p. 4). ACM. Retrieved from http://dl.acm.org/cita-
tion.cfm?id=1377971

101. 97
9 APPENDICES

102. 98

103. 99

104. 100

105. 101

106. 102

107. 103

108. 104

109. 105
Requirements Statement Priority
1. Actionability
Investigations
Allow the user to know if there is any potential high risk factor information that is miss-
ing [information>data gathering]
Rationale: It’s usually part of the consultation to gather information [P6V2C]. For high
risk factors, do I need to gather data or do tests [P2V2B]? Users want to understand if
info is missing, especially that which is known to affect risk in that condition
Design recommendation: If there is information missing for a specific risk calculator,
make it apparent to the user through the interface that the information should be gath-
ered. For example, if a user is trying to use a particular risk calculator and there is
missing data, it could be represented by empty boxes in the patient info section. In the
actual prototype the ‘order investigations’ section can be a placeholder letting the par-
ticipant know that there would be more interface to allow for investigations (a full
mock-up of this part is beyond the scope of this project). We can assume that this part
of the interface would allow adding new data points during the consultation [P6V1C]
2. Actionability
Diagnosis
Assuming the user has enough information to make a diagnosis he wants to know
what to do about risk [P2V2B]. Allow the user to move toward a diagnosis [infor-
mation>actionability]
Rationale: Nothing tells you about treatments to change the course [P5V2C]. It would
be nice to know what to do to change, such as surveillance or intervention [P2V1A].
What can I do about this for the patient [P6V1C]
Design recommendation: Similarly to Requirement Statement #1, have a part of the
interface that leads to diagnosis with a placeholder for next steps in the traditional di-
agnosis work a clinician does
3. Text
Understanding purpose of visualisation
Explicitly let the user know what the visualisation is for [user & system model mis-
match]
Rationale: Users thought the visualisation was showing co-morbidities of diabetes
[P3V1A], thought the patient had the condition already [P1V2A], did not know if the
data was from a patient or just generic/textbook data showing all risk for a particular
disease
Design recommendation: The text should be changed to support user understanding.
‘Risk of developing X’
4. Text
Patient information
The user understands data that does not contribute to the risk factor as neutral or not
contributing to risk [information>categories]
Rationale: All data is considered ‘patient information’, even the data that infers risk
[P2V1A]
Design Recommendation: Refer to ‘patient information’ that does not contribute to risk
as ‘neutral patient info’ or ‘patient info not contributing to risk’

110. 106
5. User input
Adding data to individual data points
There should be the possibility for the user to add contextual info & notes to data
points [information>supporting interpretation]
Rationale: Everyone makes sense of data in their own way. By allowing the user to
populate data points with contextual information & notes [P1V1A], the thought process
behind decisions can be documented and referred to later. Patients are individual and
therefore some will be more at risk than others [P1V1&2] when it comes to the risk vs.
benefit of side effects from intervention (or doing nothing) [P5V2C]
Design recommendation: Allow user to input notes, that are then added to a data point
6. User input
Editing risk category
If there is no/low evidence for a data point, allow the user to assign a risk category.
But, do not allow the strongly evidence based data points to be moved. Track all
changes and show if an item has been moved through the interface [information/sup-
porting interpretation & “obj” vs. “subj” data]
Rationale: ‘Solid’ data has its limitations [P1V1A]. For example, occupational history is
useful [P5C1B], but interpretation is subjective. In clinical work evidence & subjective
opinions are mixed, this tool gives an objective view of both that can be reviewed
[P6V1&2]
Design recommendation: The data points that aren’t used in validated scales can be
re-assigned to another risk category, but these changes must be tracked for later re-
view
7. Data Categories
Category order
Present categories in the order clinicians are used to looking at it in their workflow [in-
formation>categories]
Rationale: Although the categories cover relevant things [P1V1A], they are not in the
order that clinicians would be used to looking at them [P5V2C]
Design recommendation: Order by patient history, social history, family history, occu-
pational history, medication, examination data, genetic data (genetic is not so con-
crete about where it should be because only specialists are likely to understand)
8. Data Categories
Genomic information
The user must be able to understand genetic information [information>categories]
Rationale: Genetic information is difficult to interpret [P5V1B]. As it means nothing to
most people [P1V2A]
Design recommendation: Summarise genetic information on the information page, but
allow zooming into details for the specialist
9. Individual data point view
Relative risk
Let the user know how a single data point correlates to risk [P6V2C] [risk type>relative
risk]
Rationale: Relative risk is not known for all of the data points, but it would be helpful is
it was available for those points that are known [P2V2B]
Design recommendation: Display relevant patient data on classic graphs & scales
within the individual data point view. Let the user know the placement on
high/med/low in the overview is based on the relative risk of that specific measure-
ment in order to ground the perspective [P2V1A]

111. 107
10. Individual data point view
Contextual data
The user must have access to:
 Temporal factors if they are relevant (how long has he smoked, how long has
he lived here, how long did he live at his previous residence etc.)
 Metadata if it is relevant (where was this is the bone density at the hip, what
is the bone density at the spine etc. for DEXA scan)
[information>contextual data]
Rationale: If a person has only smoked for 5 a day for 2 weeks vs. 40 a day for 20
years and the 2 week patient still smokes & the 20 year stopped last week, it’s point-
less to call him a non-smoker when assessing risk [P6V1&2]. If bone density is mas-
sively low in hip but normal in spine the DEXA scan should have metadata attached to
it so the non-specialist can easily process that info [P5V1B]
Design recommendation: Include temporal factors & metadata where relevant when
zooming in on the individual data points
11. Individual data point view
Doctors notes
Allow the user to view notes about each individual data point [information>contextual
data]
Rationale: Having access to notes about to a data point would be useful for user. The
user would check notes of other specialist when looking at data they might not under-
stand [P2V2B]
Design recommendation: Have button to doctors notes in the individual data point
view which allows the user to read any notes that are attached to a specific data point
12. Individual data point view
Literature & official information portals
Allow the user to view evidence about the risk associated with each data point [infor-
mation>supporting sources]
Rationale: User wants to see all factors and be able to assess evidence behind it,
useful for things with low clinical utility [P2V2B], for example if a patient asks about e-
cigarettes [P6V2C]. Different clinicians remember risk differently, literature should be
there to be able to check [P6V2C].
Design recommendation: Have a ‘view evidence’ button that shows evidence & litera-
ture for data point
13. Individual data point view
Guidelines
Allow the user to view NICE, NCCN (for oncology) or other guidelines relevant to the
individual data point where releavant [information>supporting sources]
Rationale: NCCN guidelines for patient surveillance would be helpful [P2V2B]. NICE
guidelines for a range of options & treatments [P5V2C]
Design recommendation: Include a button that links a specific data point to the rele-
vant official guideline (simple example: what does NICE say I should do about high
blood pressure?)
14. Individual data point view
Back button
Make it easier for the user to navigate back to the overview [interface specific]
Rationale: User doesn’t know how to get back to overall risk [P2V2B]
Design recommendation: Make the back button more visually apparent/allow the user
to pinch zoom out to the overview

112. 108
15. Individual data point view
Border design
Keep the interface design clean & consistent [interface specific]
Rationale: User was surprised when the border (purple above and below data) ap-
peared [P1V2A]. Visually nice, but found it unnecessary for task and distracting
16. Individual data point view
Patient information bar
Make sure the user understands whether an individual data point contributes to risk,
protective or it neutral in the individual data point view [risk type>overall risk]
Rationale: Some most users did not comment on it, some users understood [P3V2C],
and some did not notice but mentioned the point [P6V2C] of seeing the risk severity
Design recommendation: Make the zoomed in patient information bar red & green
lines more prominent
17. Overview
Overall risk calculation
Give the user an overall risk calculation that changes if modifiable factors are changed
in the patient [P3V1&2] [interface specific]
Rationale: overall risk was missing from both visualisations, something the majority of
participants noted [P2V1&2] [P3V1&2] [P1V1&2] [P6V1&2]. Considering the disparity
of the data sources that are represented in the visualisations, there are no risk calcu-
lators that take all factors into account. A solution to this is to include existing risk cal-
culators (whichever is preferred by clinician) to display a summary number, because
in the end it’s all complex but clinicians want a summary [P1V2A]
Design recommendation: Individual conditions have their own risk calculators (i.e. risk
of diabetes in 5 years is X [P3V1A]), so a number, percentage, slider bar etc. [P2V2B]
representing overall risk should be implemented
18. Overview
Patient verification
Keep identifying information about the patient clearly visible on the overview screen
[interface specific]
Rationale: The name and photo of patient is good for verifying it’s the right person
[P5V1B] [P5V2C] [P6V2C]
19. Overview
Risk factors
Make high/medium/low risk factors visually apparent to the user [interface specific]
Rationale: It would be good to highlight risk factors that raise risk immensely [P2V2B].
Thought Visualisation 2 didn’t have high/med/low scale [P2V1A]. The user needs to
be primed to the scales to see the size of box is relevant to the scale of the issue
[P1V2A]. Not sure if size of colour represents importance in columns [P4V2B]
Design recommendation: Show risk level with different shades of colour [P6V2C] with
high risk given the highest visual priority
20. Overview
Colours
Rethink the colours used to distinguish categories [interface specific]
Rationale: Participant expects red to signify risk [P6V2C]. Some category colours
clash with colour scales in the single data point view [P1V2A]. Red-green next to each
other is bad for universal design (not colour-blind friendly) [P1V2A]

113. 109
21. Overview
Size of data point
Make it possible for the user to navigate to specific single data point view without error
[interface specific]
Rationale: Pixel area makes it hard to find and navigate to each data point [P1V2A],
this could result in errors where the user accidentally navigates to a data point he
wasn’t trying to reach
Design recommendation: Make each data point large enough to tap on a touch inter-
face (or allow pinch zoom)
22. Differentiation
Can/cannot change
These factors are of high importance to the clinician as it relates to what data points
are actionable [interface specific]
Rationale: Modifiable & unmodifiable, can change & can’t change, static & dynamic
were terms for the same concept that clinicians said were useful to the working doctor
[P3V2C] [P1V1&2] [P2V2B] [P4V1B]
Design recommendation: Give changeable risk factors the most visual priority
23. Differentiation
Clinically validated
Change ‘clinically validated’ to something that is better a better conceptual fit for clini-
cians [interface specific]
Rationale: Clinicians work with ‘soft’ & ‘hard’ data/qualitative & quantitative data, but
the term ‘clinically validated’ was poorly understood; ‘if it’s not validated why is it
there?’ [P1V2A], ‘why show factors that are not validated?’ [P4V2B]. In the words of
one participant ‘we don’t necessarily trust the data, but we still like to include it in our
calculations’ [P2V2B], with this in mind the visualisations were meant to give, what
participant 6 called ‘an objective view of evidence based & subjective data’ [P6V1&2].
Thus, we should focus on showing the data points that contribute to an evidence
based calculation of risk
Design recommendation: Have evidence based risk factors on the right, and patient
data/contextual factors on the left (but keep colour codes to show the risk factors
within patient information).
24. Filters
Search
Allow the user to search for a data point [external factors/observation notes]
Rationale: The large amount of patient data [P1V1A] may lead to particular data
points being hard to find. Some participants found it hard to find particular data points
when asked to interact with the concept
Design recommendation: Add a search bar to find specific data points