Evaluation of anything
... that is designed for external consumption,
needs to be evaluated by external agents.
Can be a front end, film, pepsi challenge…
or a piece of writing.
Is the work fit for purpose?
Should be a no-brainer, but...
Reasons not to evaluate
It is a hassle
You lay your work open to external opinion
Alterations will mean more work
Some people think it is better just to assume it’s
great and stick your head in the sand...
Ways of evaluating
Can be difficult to apply
Use them in initial design
Where do you get an unbiased expert from?
Best but hard work
Real users - usability testing
A thing for them to use
They have to actually use it
A test; questions, and results
A thing to use
Screen based (Photoshop, Flash)
Single track/walk through
Wizard of Oz
Photocopoier, spreadsheet, tipex, scissors,
By hand or printouts from layout program
Early and fast = incredible savings in effort
Good can be small and fast
Big and slow can be bad. “Our site needs
redesigning its rubbish, get 10 people into the
usability lab and conduct in depth analyses of
whats wrong with it…”
Here’s an idea; hand sketch it, small group,
‘where would you click for more information?’.
They have to use it!
Not about just asking a flatmate ‘is it user
friendly?’ They will say ‘Yeah, what’s on TV?’.
‘Do you want to stay my friend, and does this
look like I’ve put some work into it?’
Use it… Task oriented
Is it easy to use? Is it quick to use ?
How long does it take the user get an average
ticket if they already know how to use the
machine? When new users use it for the first
time can they all get the ticket they require?
Observations or questions
‘How many seconds to click it?’
‘Does the site have events information?’
Can do manipulations with the data
Vey focussed and might be measuring the
Interviews, chatty or open questions
Data is less maleable
Questions have to be:
What to ask, how to ask it
What to ask - Life on Mars
‘Well, what do you think; is it easy and fast to
Mars probe. ‘Well, what do you think; is there
life on Mars?’
Break the big quesitons down into the detailed
quesitons that we can actually get answers to.
Life on Mars...
The National Research Council panel
nicknamed the "weird life" committee. The
group worries that scientists may be too Earthcentric when looking for extraterrestrial life. The
problem for scientists is that "you only find what
you're looking for," said Penn State University
geosciences professor Katherine Freeman
How to ask it?
Can be very tricky
The Elderberry Experiments 60’s (more later)
Societal and personal pressures
Your best friend likes your design - its the
Bradley effect (Tom Bradley 1982 mayor)
‘Do you mind if I smoke?’ Vs ‘Would you prefer
it if I didn’t smoke’
Scots referendum. “Do you agree that…”
Capturing results - which method
Method has to be appropriate
Observation doing a task
Metrics, talk through
Questions in a form
Questions in an interview
Video them and ask them ‘did you see the help
button on the page?’.... ‘Yes’
Simple closed questions use fill in forms with
What do you think of the help function?
Did you use the help function?
I found the help function
Be very careful with multiple choices
Audio and video
Transcription/analysis is VERY time consuming
Audio for interviews
Video for visual oriented investigations
Already a large amount of good practice
Processing the recordings
… and competitor analysis
… is analysing your competitors to see how
you stack up against them
… is looking at what others are doing
it’s not about copy and pasting bits here and
there to create your own product. Not the
Fast and small and many
Slides on slideshare.net/lonbarfield