Presented 5/11/17 @LOCO_UX by @jkooda of @liminaUX
This talk covers the anatomy of a UX Eval, how to use it as a business development tool, and how to ensure you have a logical and most importantly beneficial return on your client's investment.
The Power of
The UX Evaluation
a LOCO | UX Meetup
A FEW SHORT WORDS
LOCO | UX Community Mission
Build a network of UX practitioners local to the Longmont/Boulder
area for the purpose of:
• Access to opportunities (seeking, hiring, colabos)
• Sharing Ideas, Tools, Methods, Techniques
• Building community for mutual enhancement/benefit
• each other
• local businesses
• Longmont municipality
Who Tweets During Meetups?
@jkooda of @liminaUX at @loco_ux
enabling a person to discover or learn something for
"a “hands-on” or interactive heuristic approach to learning"
a heuristic process or method.
Is a usability engineering method for finding usability
problems in a user interface design, thereby making
them addressable and solvable.
It involves a small set of expert evaluators who examine
the interface and assess its compliance with
“heuristics,” or recognized usability principles.
10 Usability Heuristic Principles
Visibility of system status
Match between system and the
User control and freedom
Consistency and standards
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Helps users recognize, diagnose,
and recover from errors
Help and documentation
Examples in the Wild: http://designingwebinterfaces.com/6-tips-for-a-great-flex-ux-part-5
Donald A. Norman
A heuristic evaluation should not
replace usability testing.
Although the heuristics relate to criteria that affect
your site’s usability, the issues identified in a heuristic
evaluation are different than those found in a
• Provides quick and relatively inexpensive (cheap)
• Systematic and Intuitive, applying a set of predefined rules/heuristics
• Requires very little planning
• Can be used early in the design process
• Does not require a finished interface
• Provides corrective measures
• Can be used together with other usability testing methods
• Can help identify and prioritize issues in need of further usability
• Requires knowledge and experience to apply the heuristics effectively
• Trained usability experts are sometimes hard to find and can be
• Requires multiple experts and aggregate their results
• The evaluation may identify more minor issues and fewer major issues
• Tends to focus on problems rather than solutions
1. Set Up The Team
2. Agree on Evaluation Criteria
3. Conduct Cognitive Walk Through
4. Analyze Interface & Record Issues
5. Internal Review
6. Present to Client
7. … Next steps & business development
6 MAJOR PARTS
Rows = 19 evaluators
Column = 16 usability problems
= Problem found
Economics of the Evaluation Team
1 5 10 15
Number of Evaluators
Strength in Numbers
1 5 10 15
Number of Evaluators
1.Emergence over authority – Talks about the shift in authority from the top-down to the bottom-up.
2.Pull over push – Another example of bottom-up demand. User-driven instead of mass-marketing.
Think on-demand instead of broadcast.
3.Compasses over maps – Methodology not mission. Teach a man to fish. A play for Seymour Papert’s
4.Risk over safety – When the cost of innovation is low, like it is now, it’s easier take more risks and try
more things. Negroponte famous MediaLab motto was, “Demo or Die”, Ito says “Deploy.”
5.Disobedience over compliance – “You don’t win a Nobel Prize by doing what you’re told” (Joi Ito). Ask
forgiveness, not permission.
6.Practice over theory – “In theory there is no difference between theory and practice. In practice there
is” (Yogi Berra). Both should inform one other.
7.Diversity over ability – Crowdsourcing (a word Howe coined) can solve
some problems that dedicated professions can’t solve internally.
8.Resilience over strength – They quote the old adage that an oak tree is strong, but can be blown over
in a storm; while the weaker but flexible willow bends with it and can survive. Akin to what Angela
Duckworth popularized as “grit” in her great book with the same title.
9.Systems over objects – The value of interdisciplinary networking.
• Ease of Use
• Taxonomy /
• Interface Elements
• Error Handling
• Access to Help
• Page Layout
• Color Use
• Visual Language
• Data Visualization
BREAKDOWN OF RESPONSIBILITY
USER RESEARCHER INTERACTION DESIGNER VISUAL DESIGNER
Getting Domain Expertise
BUSINESS REPRESENTATIVES — Wrangle business objectives and understand
opportunities for the business. Understand how to make or save money for the
business. Understand how the design meets the needs of the product domain and its
PROJECT MANAGER — Ensure the client can implement this design in time and
when/what the project milestones are.
TECHNICAL EXPERT — Understand technical framework (opportunities and limitations).
Validate the client can implement the product recommendations (feasibility / time
YOUR TEAM - provides the domain expertise in “USER EXPERIENCE &
USABILITY”, they provide industry domain expertise.
Plan to evaluate all key
features that differentiate this
product from it’s competitors.
Ask who else is doing this, is
any one doing this better?
KEY TASKS /
Plan to evaluate all key
workflows in the system.
Unless problematic, avoid
evaluating common tasks (ex:
Understand what metrics are
important to the business:
(ex. time to completion, task
completion ratios, user
BE A SPONGE:
• Take this opportunity to understand the product from the business and user perspective.
• Remain objective, do not adopt or project your cognitive bias.
• Take screenshots and where possible, record the session (voice and screen)
Draft Run Through
GAP ANALYSIS – Any areas need more
attention? Use your KPI’s, Key Workflows, Key
INTERIM CLIENT CHECKPOINT – Depending
on confidence and success, it’s always a good
idea to check in with the client around the
60% complete mark
ORGANIZE FINDINGS — Look across what
everyone has analyzed, group logically (by
issue category or functional areas – avoid
redundancy (cite instances and
DRIVE VALUE – Without giving up free work,
provide broad recommendations an cite
examples of successful implementations.
ASSIGN PRESENTERS / NOTE TAKERS —
Depending on skills mix you may want to mix
it up, unless you’re confident with one
ESTIMATE PRESENTATION TIME — These
reports can be dense and packed with rich
discussion, you will need to account for extra
time per issue.
BE SENSITIVE TO POLITICAL /
• Always start on a positive note it’s important to
acknowledge where they hit the mark and
showed innovative, unique and successful
• Often times the interfaces you’re evaluating
represent a significant amount of commitment,
time, and/or cost investment on your client’s
• Realize that every observation can be perceived
as a personal critique by any number of client
BREAK THE NEWS IN THE BEST LIGHT
• Your perspective represents an objective 3rd
party observations that should not be charged
with subjectivity and aesthetic/technical
• This is particularly important for the visual
• Your observations should be supported by
empirical evidence and sound solutions and
• Make room for conversations, check the pulse of
how each observation and recommendation is
YOU MAY HAVE CLARIFYING QUESTIONS
• Make sure you’re taking time to validate your observations and recommendations
THIS IS NOT A PASSIVE SESSION
• Create opportunity for the client to articulate priority and effort levels against your
observations and recommendations (H, M, L) this will be useful in backlog
• You are proxy for the user, be sure to advocate for the usability impact of resolving or
deprioritizing a fix, they are relying on you to be the expert
• Continue to articulate your recommendations and solutions in terms of KPI’s and what
the would be most valuable to the business without negatively impacting ease of use
and user value.
IF YOU’VE DONE YOUR JOB…
• The client would have agreed with 60%-80% of your findings
• They will already be thinking about low handing fruit and how to parse the
enhancement into near term sprints and long term release cycles – this is your
opportunity to step up.
THE EVALUATION REPORT IS THE BEGINNING, NOT THE END
• You now own the issue log and recommendations with client feedback
• Ranked by: Client Priority, User Benefit, Technical Complexity…
• This is the making of what we call a Feature Matrix: a useful tool for UX
prioritization, sprint planning and backlog management
• Catalogue UX
requirements into swim-
• Assign each cell with User
Benefit, Business Benefit
and Technical Complexity
• Use this information to build
the priority ranking for
sprint planning and backlog
POWER FOR YOU
YOUR BUSINESS DEVELOPMENT TOOL FOR UX WORK
Brings you top of mind for follow on detailed design contract work
Clearly articulates the value of UX in terms that matter to the client
Wets the appetite of the client without giving up detailed design work
Strong business planning tool – showing UX as a strategic practice
POWER FOR YOUR CLIENT
In most cases your observations and recommendations can dovetail
into their existing development plan – the client can yield immediate
value from the report
For a relatively low cost, the client has gained critical external
feedback for near and long term product enhancements – outputs
from the report are typically folded into long term development plans