A run through how I execute collaborative usability testing in an agile development process. Covering the operational detail, and tips to get stakeholders on board. Delivered at Drupalcamp Scotland, 7 November 2015

1. Usability testing in an agile
development process
How our approach to testing
might work for you too…
Neil Allison
University of Edinburgh
Drupalcamp Scotland
7 November 2015

2. This presentation isn’t about Agile
• It’s about regular, rapid, inclusive usability
testing with minimal overheads
• It just so happens that the agile process we
used to develop the new University CMS
(EdWeb) forced me to work this way
– It can work for you regardless

3. Usability testing
It’s easy, right?
http://bit.ly/krug-test-demo

4. What’s challenging
• Getting the go ahead to use
time & money on usability testing
• Getting colleagues & customers to
take on board what you uncover
• Getting fixes to problems implemented
(Why usability problems go unfixed: http://bit.ly/LvrGoq)

5. Unless you’re…
Flickr credit: randychiu

6. My challenges as UX Lead for EdWeb
• It’s not a formal role in Information Services
• Misconception that it’s the UX Lead’s job to
‘decide what’s usable’
• Team is too close to the product,
with not enough exposure to CMS users
• Striking a balance between delivering new
functionality and improving what we have

7. So what did we do?
1. Get the right people in a room
2. Watch a small number of short sessions
with users doing something
3. Prioritise the issues they see
4. Collaboratively consolidate their priority lists
5. Agree actions for usability issues
6. Repeat every few weeks

8. Who are the right people?
• Everyone with a stake in
the product
– No exceptions
http://bit.ly/1I1lZfQ
“Have you had your
recommended dose
of research?”

9. What did we watch?
• Real CMS users doing
real tasks
• Facilitated usability
testing sessions
• Focus of testing
agreed
collaboratively in
team
“Research shows that teams
make better services when
everyone on a project team
observes users first hand.”
http://bit.ly/1I1rlYI

11. How many did we watch?
“The most striking truth of the curve is that
zero users give zero insights.”
• As many as you can fit into the time you have
(so probably not very many)
http://bit.ly/1vQ7eHD

12. How did we prioritise?
“Running a usability test has been compared
with taking a drink from a fire hydrant…”
• Rocket Surgery template:
1. Individual notes while observing
2. Distil to 3 issues after each participant

14. http://bit.ly/1I1mCWW
“If you prioritise usability problems using
'gut feel' or intuition, you run the risk
of being exposed as a fraud…”
How did we consolidate?

16. Then what?
• Usability issues prioritised, not solutions
• Agree actions based on:
– Is a solution “obvious”?
– Is there an easy development solution?
– Is there an alternative to development?

17. Minor
issue for
users
Major
issue for
users
Easy
solution
available
No easy
solution
available
Prioritisation matrix

18. Recap: our process for EdWeb
In advance
• Agree test focus with team
• Write and pilot test script
• Recruit 3 participants to
turn up on the day
On the day
• 3 sessions:
– 30 minutes max
– 15 minutes between
• Observers prioritise their
notes between sessions
• Final 30(ish) minutes spent
prioritising top observations
& agreeing actions

19. Alternatives to live testing
• Record in advance if there’s just you
– Too stressful to run usability tests and facilitate
the observation group at same time
• Use a remote testing service
– www.usertesting.com
– www.whatusersdo.com
– There are others…

20. Top tips
• Participants
– A pool of volunteers really helps recruitment
– Krug – “Recruit loosely, grade on a curve”
– Reminders the day before
– Have an emergency stand-in available
• Do whatever it takes to get observers in the room
– Start over lunch
– Supply refreshments
– Bribery, favours, threats…
• Be well organised so you don’t waste anyone’s time
– Test everything before hand
• Stick to the process and schedule (particularly in the final recap)
– It’s easy to digress when you’ve all seen so much
• Encourage collective reflection on the session
– Admitting usefulness is first step to getting observers to turn up next time
– Have the next one scheduled ASAP

21. What was good about our process
For the team
• Closer to our CMS users –
immediate impact
• Shared insight & experience
• Ownership of the priority
issues
– What to fix immediately
– What we can live with that
we thought was a problem
For me
• Process keeps set up and
organisation of session to a
minimum
• No report writing
• Moves the culture of the
team on, emphasising CMS
usability on the agenda

22. What we need to do better
• How we reduce usability problems occurring
in the system in the first place
– Developer time at a premium
– Limited time for collaborative forward planning
• Getting more of the right people in the room
– For longer and more frequently

23. Everything you need
• Steve Krug’s Rocket Surgery
resources:
http://bit.ly/1I1muXo
• David Travis’ prioritisation flowchart
http://bit.ly/1I1mCWW
• My blog article on this process:
http://bit.ly/uoe-agile-usability

24. Thank you
Questions?
Neil.Allison@ed.ac.uk
Website Programme UX Manager
University of Edinburgh
Twitter: @usabilityed