UX Evaluation Design and Analysis

Preamble Science and Generalisation Break! Evaluation Design and Analysis Practical Ethical Procedures Wrapping Up
The User Experience
from 30,000ft
#comp33512
Week 07 – Lectures 13/14
Simon Harper
University of Manchester
Semester 2 – 2013/14
last update: March 12, 2014
The User Experience from 30,000ft 1 / 24

UX Pop Quiz
1. What are the pros and cons of gamiﬁcation?
2. How could you include social / group dynamics into your
system?
3. How can you enhance the users perception of fun?
4. What is the ‘skeptic’ view of Gamiﬁcation?
5. List a principle and describe it.
The User Experience from 30,000ft Preamble 2 / 24

Hold On!
Where’s Aﬀective Computing?
...skipping ahead one Week!

Designing Your Evaluations
1. Badly Designed = Incorrect Analysis;

2. Incorrect Analysis = Incorrect Conclusions; which means

3. Success of your Interventions in Doubt.

3. Success of your Interventions in Doubt.
This Means
If evaluations are not designed correctly the previous ≈207 pages
of the course notes have been, to a large extent, pointless.

Science and Generalisation
Inductive reasoning
Evaluates and then applies to the general ‘population’
abstractions of observations of individual instances of members of
the same population
The User Experience from 30,000ft Science and Generalisation 5 / 24

Inductive reasoning
the same population
Deductive reasoning
Evaluates a set of premises which then necessitate a conclusion –
for example: {(1) Herbivores only eat plant matter; (2) All
vegetables contain only plant matter; (3) All cows are herbivores}
→ Therefore, vegetables are a suitable food source for Cows.

Inductive reasoning
the same population
Deductive reasoning
1. Therefore, the conclusion must be true provided that the
premises are true;
2. Note that you could not say ‘Therefore, all cows eat
vegetables’ because fruit also contains only plant matter; as
do grass and trees.

Scientific Bedrock
To Be scientific,
A method of inquiry must be based on the gathering of
observable, empirical and measurable evidence, and be subject to
specific principles of reasoning.

Scientiﬁc Bedrock
Figure: ‘The Scientiﬁc Method’

Scientiﬁc Bedrock
An Inductive Example
1. Firstly, we create an hypothesis which, in the best case, cannot
be otherwise interpreted and is ‘refutable’; for example we might
make the statement ‘all swans are white’. In this case we may
have travelled widely and tried to observe swans in every country
and continent in an attempt to support our hypothesis.

Scientiﬁc Bedrock
2. While, we may be able to amass many observations of white
swans we must also realise that a statement must be refutable. If
the hypothesis remains intact it must be correct; in our example
we may try to observe every swan that exists in, say, the UK, or
Europe, or the Americas, which is not white.

Scientiﬁc Bedrock
3. However, one instance of an observation of a non-white swan
will disapprove our hypothesis; in this case when we arrive in
Australia we discover a black swan, in this case we can see all
swans are not white and our hypothesis is found to be incorrect.

Scientiﬁc Bedrock
Many debates regarding the question of whether inductive
reasoning leads to truth;
We can make some inductive leaps if they are based on good
science;
These leaps may not be absolutely accurate; but
May well assist us in our understanding; in the
UX domain we use mathematical (statistical) methods to
help us understand these points.

Mathematical (Statistical) Methods
Generalise results to enable us to say something about the
wider population; so
We use well formed and tested statistical tests;
Which enables use to mathematically generalise to a
population; this is called,
External Validity.

Mathematical (Statistical) Methods
Generalise results to enable us to say something about the
wider population; so
We use well formed and tested statistical tests;
Which enables use to mathematically generalise to a
population; this is called,
External Validity.
No 100% Certainty
All we have is a level of conﬁdence in how a particular test relates
to the population, and therefore how useful the knowledge
generated from it really is.

Variables / UX
Behavioural: Equated to the user;
Stimulus: Equated to the interface or the computer
system;
Observable Response: the thing we measure to understand if
there is a beneﬁt after we have
manipulated the stimulus; and
Subject: Factors such as age, weight, gender.
Independent Variable: The thing that we manipulate – the lower
the number of independent variables, the more
conﬁdent we can be about the data collected and
the results of the analysis; and
Dependent Variable: The thing that we measure.

Measuring Dependent Variables
Nominal Scale: Which denotes identity (Categorical);
Ordinal Scale: Which denotes identity and magnitude;
Interval Scale: Denotes identity, magnitude and has the beneﬁt
of equal intervals; and
Ratio Scale: Which has the positive properties of the three we
have already seen as well as a true zero point.

Measuring Dependent Variables
Nominal Scale: Which denotes identity (Categorical);
Ordinal Scale: Which denotes identity and magnitude;
Interval Scale: Denotes identity, magnitude and has the beneﬁt
of equal intervals; and
Ratio Scale: Which has the positive properties of the three we
have already seen as well as a true zero point.
Variables, and their measurement, are important.
They inform the experimental design process and the kind of
analysis that will be possible once the data has been collected.

Hypothesis Testing
Null Hypothesis: Which dictates that there is no difference
between two conditions beyond chance
differences; or
Hypothesis: Which dictates there is a difference and
supports the hypothesis proposed.

Hypothesis Testing
diﬀerences; or
Strong and Weak
A hypothesis must be ‘strong’ to be testable.

Hypothesis Testing
diﬀerences; or
Strong and Weak
A hypothesis must be ‘strong’ to be testable.
Nothing is Ever Proved
Hypotheses are supported or disproved - NOT ever proved (in
empirical work).. Why?

Let’s Have a Break!
Back in 10 Minutes!
Come see me now if you have
Questions Regarding this Lecture!
The User Experience from 30,000ft Break! 12 / 24

Hypothesis Testing
http://www.youtube.com/watch?v=vKA4w2O61Xo
The User Experience from 30,000ft Break! 13 / 24

Evaluation Design and Analysis
Experimental Design;
Data Collection and Tools;
Data Analysis; mostly
Statistical Analysis.
The User Experience from 30,000ft Evaluation Design and Analysis 14 / 24

Evaluation Design and Analysis
Descriptive Statistics;
Inferential Statistics.
Internal Validity;
External Validity; and
Confounding Variables.

Participants
Simple Random Sampling Probabilistic — Simple random
sampling equates to drawing balls at a tom-bola. The selection of
the ﬁrst has no bearing, and is fully independent of, the second or
the third, and so forth. This is often accomplished in the real
world by the use of random number tables or, with the advent of
computer technology, by random number generators;
Systematic Sampling Probabilistic — Systematic samples are a
variation of random sampling whereby each possible participant is
allocated a number, with participants being selected based on
some systematic algorithm. In the real world we may list
participants numbering them from, say, one to three hundred and
picking every seventh participant, for instance;

Participants
Stratified Sampling Probabilistic — Stratified samples are used
to reduce the normal sampling variation that is often introduced
in random sampling methods. This means that certain aspects of
the sample may become apparent as that sample is selected. In
this case, subsequent samples are selected based on these
characteristics, this means that a sample can be produced that is
more likely to look like the total population than a random sample;
Multistage Sampling Probabilistic — Multistage sampling is a
strategy for linking populations to some kind of grouping. If a
sample was drawn from, say, the U. of Manchester then this may
not be representative of all universities. In this case, multistage
sampling could be used whereby a random sample is drawn from
multiple different universities independently and then integrated.
In this way we can ensure the generalisability of the findings; and

Participants
Quota Sampling Non-Probabilistic — Almost all
non-governmental polling groups or market research companies
rely heavily on non-probability sampling methods; the most
accurate of these is seen to be quota based sampling. Here, a
certain demographic proﬁle is used to drive the selection process,
with participants often approached on the street. In this case, a
certain number of participants are selected, based on each point
in the demographic proﬁle, to ensure that an accurate
cross-section of the population are selected;
Snowball Sampling Non-Probabilistic — The process of snowball
sampling is much like asking your participants to nominate
another person with the same trait as them.

Participants
Convenience Sampling Non-Probabilistic — The participants
are selected just because they are easiest to recruit for the study
and the UX’er did not consider selecting participants that are
representative of the entire population.
Judgmental Sampling Non-Probabilistic — This type of
sampling technique is also known as purposive sampling and
authoritative sampling. Purposive sampling is used in cases where
the specialty of an authority can select a more representative
sample that can bring more accurate results than by using other
probability sampling techniques.

Evaluation‘++’
Single Group, Post Test;
Single Group, Pre-Test and Post-Test;
Natural Control Group, Pre-Test and Post-Test;
Randomised Control Group, Pre-Test and Post-Test;
Within Subjects; but there are
Others.

Practical Ethical Procedures
The Ethical Process
A critical component of good evaluation design because it
encourages the UX specialist to focus on the methodology and
the analysis techniques to be used within that methodology.
The User Experience from 30,000ft Practical Ethical Procedures 18 / 24

Organisations
The American Psychological Association’s (APA), ‘Ethical
Principles of Psychologists and Code of Conduct’;
The United States Public Health Service Act (Title 45, Part
46, Appendix B), ‘Protection of Human Subjects’;
The Belmont Report, ‘Ethical Principles and Guidelines for
the Protection of Human Subjects of Research’;
The Council of International Organisations of Medical
Sciences, ‘International Ethical Guidelines for Epidemiological
Studies’; and ﬁnally
The World Medical Association’s, ‘Declaration of Helsinki –
Ethical Principles for Medical Research Involving Human
Subjects’.

in Brief...
About You...
Competence: Keep up to date, know your limitations, ask for
advice;
Integrity: Have no axe to grind, or desired outcome; and
Science: Follow the Scientiﬁc Method.

in Brief...
About Them...
Respect: Assess you participants autonomy and capability
of self-determination, treat participants as equals,
ensure their welfare;
Benefits: Maximising benefits and minimising possible harms
according to your best judgement, seek advice
from your organisations ethics committee;
Justice: Research should be undertaken with participants
who will benefit from the results of that research;
and
Trust: Maintain trust, anonymity, confidentiality and
privacy, ensure participants fully understand their
roles and responsibilities and those of the
experimenter.

in Brief...
About Us...
Responsibility: You have a duty of care, not only to your
participants, but also to the community from
which they are drawn, and your own community of
practice.

Discussion Topics Coursework # 3
‘Voice Loops as Cooperative Aids in Space Shuttle Mission
Control’ (10 Marks) – this paper shows just how far UX and the
techniques which it inherits from human computer interaction can
go. We are mainly concerned with systems and objects which are
purely commercial, however, in this case failures in the human
interface can have serious consequences for a real-time mission,
including the loss of the vehicle. Further, these kind of UX
techniques can also be found in other critical interface
components such as those controlling nuclear power stations or
ﬂy-by-wire aircraft.
Jennifer C. Watts, David D. Woods, James M. Corban, Emily S. Patterson, Ronald L. Kerr, and LaDessa C.
Hicks., Voice loops as cooperative aids in space shuttle mission control., In Proceedings of the 1996 ACM
conference on Computer supported cooperative work, CSCW ’96, pages 48–56, New York, NY, USA, 1996.
ACM., ISBN 0-89791-765-0., http://doi.acm.org/10.1145/240080.240188., URL
http://doi.acm.org/10.1145/240080.240188.
The User Experience from 30,000ft Wrapping Up 21 / 24

Pop Quiz for next (logical) week...
1. What is the scientiﬁc method and why is it important?
2. What do we mean by internal and external validity?
3. What is the single most important reason for having a set of
ethical procedures?
4. What are the eight key ethical principles (give a brief
rationale for each)?
5. Why is conforming to scientiﬁc principles key to good ethical
designs?

To Do for next week...
1. Pop Quiz, Discuss Next Week; and
2. Read your notes!

Any Questions?
Simon Harper 2.44 Kilburn Building
0161 275 0599 (OR x50599)
simon.harper@manchester.ac.uk
Oﬃce Hours: Friday 14:00–18:00
Figure: ‘Moving the Order’

UX Evaluation Design and Analysis

Recommended

Recommended

More Related Content

Similar to UX Evaluation Design and Analysis

Similar to UX Evaluation Design and Analysis (20)

More from Simon Harper

More from Simon Harper (16)

Recently uploaded

Recently uploaded (20)

UX Evaluation Design and Analysis