An outline of the usability testing process used within the agile process being used to develop a new Content Management System (CMS) for the University of Edinburgh. Presented to an audience of University staff 17 December 2014, followed by a usability testing and issue prioritisation session that demonstrated the methodology.

1. Making usability testing agile
How our approach to testing in the
EdWeb CMS development project
might work for you too…
Neil Allison
University Website Programme
UX Interest Group 17 Dec 2014

2. This presentation isn’t about Agile
• It’s about regular, rapid, inclusive usability
testing with minimal overheads
• It just so happens that the agile process we’re
using to develop the new University CMS
(EdWeb) forced me to work this way
– It can work for you regardless

3. Usability testing
It’s easy, right?

4. What’s challenging
• Getting the go ahead to use
your time on usability testing
• Getting colleagues to take on board
what you uncover
• Getting fixes to problems implemented
(Why usability problems go unfixed: http://bit.ly/LvrGoq)

5. Unless you’re…
Flickr credit: randychiu

6. My challenges as UX Lead on EdWeb
• It’s not a formal role in Information Services
• Misconception that it’s the UX Lead’s job to
‘decide what’s usable’
• Team is too close to the product,
with not enough exposure to CMS users
• Striking a balance between delivering new
functionality and improving what we have

7. So what do we do?
1. Get the right people in a room
2. Watch a small number of short sessions
with users doing something
3. Prioritise the issues they see
4. Collaboratively consolidate their priority lists
5. Agree actions for usability issues
6. Repeat every few weeks

8. Who are the right people?
• Everyone with a stake in
the product
– No exceptions
http://bit.ly/1I1lZfQ
“Have you had your
recommended dose
of research?”

9. What do they watch?
• Real CMS users doing
real tasks
• Facilitated usability
testing sessions
• Focus of testing
agreed
collaboratively in
team
“Research shows that teams
make better services when
everyone on a project team
observes users first hand.”
http://bit.ly/1I1rlYI

10. How many do they watch?
“The most striking truth of the curve is that
zero users give zero insights.”
• As many as you can fit into the time you have
(so probably not very many)
http://bit.ly/1vQ7eHD

11. How do they prioritise?
“Running a usability test has been compared
with taking a drink from a fire hydrant…”
• Rocket Surgery template:
1. Individual notes while observing
2. Distil to 3 issues after each participant

12. How do we consolidate?
“If you prioritise usability problems using
'gut feel' or intuition, you run the risk
of being exposed as a fraud…”
http://bit.ly/1I1mCWW

13. Then what?
• Usability issues prioritised, not solutions
• Agree actions based on:
– Is the solution “obvious”?
– Is there an easy development solution?
– Is there an alternative to development?

14. Recap: our process for EdWeb
In advance
• Agree test focus with team
• Write and pilot test script
• Recruit 3 participants to
turn up on the day
On the day
• 3 sessions:
– 30 minutes max
– 15 minutes between
• Observers prioritise their
notes between sessions
• Final 30(ish) minutes spent
prioritising top observations
& agreeing actions

15. Top tips
• Participants
– A pool of volunteers really helps recruitment
– Krug – “Recruit loosely, grade on a curve”
– Reminders the day before
– Have an emergency stand-in prepared
• Do whatever it takes to get observers in the room
– Start over lunch break
– Supply refreshments
– Bribery, favours, threats…
• Be well organised so you don’t waste anyone’s time
– Test everything before hand
• Stick to the process and schedule (particularly in the final recap)
– It’s easy to digress when you’ve all seen so much
• Encourage collective reflection on the session
– Admitting usefulness is first step to getting observers to turn up next time
– Have the next one scheduled ASAP

16. What’s good about our process
For the team
• Closer to our CMS users –
immediate impact
• Shared insight & experience
• Ownership of the priority
issues
– What to fix immediately
– What we can live with that
we thought was a problem
For me
• Process keeps set up and
organisation of session to a
minimum
• No report writing
• Moves the culture of the
team on, emphasising CMS
usability on the agenda

17. What we need to do better
• How we reduce usability problems occurring
in the system in the first place
– Developer time at a premium
– Limited time for collaborative forward planning
• Getting more of the right people in the room
– For longer and more frequently

18. Everything you need
• Steve Krug’s Rocket Surgery resources:
http://bit.ly/1I1muXo
• David Travis’ prioritisation flowchart
http://bit.ly/1I1mCWW

19. Now let’s try it together…
• 3 participants
– 5-10 minute break between each to review notes and
prioritise top 3 issues
• On your table consolidate your issues into a
master list
– Use flowchart to propose severity
• Once round the room to feed issues & priority
back to the EdWeb development team

20. Thank you
Questions?
Neil.Allison@ed.ac.uk
UX Manager
University Website Programme
Twitter: @usabilityed