This document discusses usability modeling and measurement. It defines usability and explains why it is important, especially for web and mobile applications. It describes various usability models and standards. The document outlines how to develop a usability model for an organization by defining measurable attributes and characteristics. It also discusses different usability evaluation methods like surveys, heuristic evaluation, and logging. The key steps are to define a model, identify attributes to measure, collect data, analyze the results, and refine measurements. Taking these steps allows an organization to systematically evaluate and improve usability.

1. Usability Modeling and Measurement
Philip Lew
www.xbosoft.com

2. Agenda
• Introduction
– The importance of usability
– Specifics for the web / mobile
• What is Usability?
• Usability Modeling and Measurements
• Case study
• Conclusion
• Next Steps

3. Importance of Usability

4. Why is Usability Important?
• Usability is important especially for web
applications (SaaS, and websites)
– And now especially mobile applications
• Most prevalent development delivery model
today
• Without good usability:
– Users will leave the applications
– For mobile, if they can’t learn in 30 seconds,
they won’t come back

5. Web and Mobile have Changed the
Terrain
• Business models
have changed
– Instead of paying
upfront and
‘owning’ the
software
– Pay as you go, pay
by subscription
• Behavior and
expectations have
changed

6. Many Alternatives
• Depending on the
type of application
– Users could leave
and go elsewhere
– Simply not use your
mobile app (if only
providing
alternative access
to a main
application)

7. Thin Client Deployment
Most Prevalent Development Delivery Model Today
• Mobile and Cloud are singing the same song
• Service delivery versus product delivery

8. What is Usability
In relation to quality
In relation to user experience

9. General Thoughts on Usability
• Understandability
• Learnability
• Operability
• Attractiveness
• Navigation
• Responsiveness-performance
• Efficiency

10. Usability in Quality Standards
• How is usability Standard Definition
defined? ISO 9126-1 The capability of the
software product to be
– Standard (2000)
understood, learned, used,
definitions and attractive to the user,
when used under specified
– Many others conditions.
who say ISO 9241-11 The extent to which a
(1998) product can be used by
similar specified users to achieve
specified goals with
things effectiveness, efficiency,
• Usability and satisfaction in a
specified context of use.
– Component IEEE 600.12 The ease with which a user
can learn to operate,
of quality as (1990) prepare inputs for, and
listed in interpret outputs of a
system or component.
many
standards

11. Usability Model Comparisons
Seffah [33]

12. Current Research / Usability Standards
Kappel et al., The Discipline of
Systematic Development of Web
Applications, 2003, John Wiley
and Sons.

13. Current Research – Usability Standards
Quality
Satisfaction Usability
ISO 9241-11
ISO
9126-1
IEEE 610
ISO
IEEE XXXY 25010

14. Bigger Picture Quality
From ISO point of view
Usability
Usability
CMMI ISO 25010

15. Usability as a Key Characteristic of
Product Quality
Source: ISO 25010

16. What is Usability-Effect of the
Software Product
Degree to which specified
users can achieve specified
goals with effectiveness,
efficiency and satisfaction in a
specified context of use.
Source: ISO 25010

17. What is Usability-Effect
In Actual Usage
• Effectiveness
– The degree to which specified users can achieve specified goals with
accuracy and completeness in a specified context of use.
• Efficiency
– The degree to which specified users expend appropriate amounts of
resources in relation to the effectiveness achieved in a specified
context of use.
– NOTE Relevant resources can include time to complete the task,
materials, or the financial cost of usage.
• Satisfaction
– The degree to which users are satisfied in a specified context of use.
Satisfaction is further subdivided into sub-characteristics:
• Likability (cognitive satisfaction)
• Pleasure (emotional satisfaction)
• Comfort (physical satisfaction)
• Trust
Source: ISO 25010

18. Usability in Actual Usage
• User role specified users
• Objective
specified goals
• Task
• Environment
specified context of use
• Domain
What else can you think
•…
of?

19. Defining Usability For Your Organization
Quality
Usability
Characteristic 1 Characteristic 2 Characteristic n
Subcharacteristic 1 Subcharacteristic 2 Subcharacteristic n
Attribute 1 Attribute 2 Attribute 3 Attribute 1
• Attributes expressed hierarchically
• Any number of sub-levels is OK
• Achieving the sub-attributes=achieving the high level
attribute -measurable

20. Let’s Define Usability
From the Product Viewpoint
Usability
Characteristic 1
Navigation Understandable
Characteristic 2 Characteristic n
Subcharacteristic 1
Control Stability Subcharacteristic 2
Previous-Next Subcharacteristic n
Attribute 1
Home Location Attribute 2
Save Location Ease of finding Attribute n

21. Bigger Picture on Usability
Usability
Usability
• Can be measured from the design point of
view or of the product
• Can be measured ‘in-use’ with real users

22. Defining Usability from an
Effect-Real usage Point of View
Usability
Quality
Effectiveness Efficiency Satisfaction Characteristic n
Accuracy Completeness Subcharacteristic n
Errors Attribute 2 Attribute 3 Attribute 1

23. Other Potential Attributes
Measurements for ‘in use’ Usability
• Effectiveness
– Completion rates
– Error rate
– Help usage
• Efficiency
– Task time
– Backtracking
• Learnability
– Learning rate
– Task time deviation

24. Developing a Combined
Usability Model
Measured ‘in-use’
Usability
Measurable and
quantifiable factors
User Behavior and
Activity
Application Design and
Environment

25. Example of Product Quality Model
with Attributes
External Quality Requirements (for Shopping Cart Entity)
1 Usability (Operability in ISO 25010)
1.1 Understandability
1.1.1 Icon/label ease to be recognized
1.1.2 Information grouping cohesiveness
1.2 Learnability
1.2.1 ………………………………………………………..
1.3 Operability
1.3.1 Control permanence
1.3.2 Expected behaviour of Controls
2 Content Quality
2.1 Content Suitability
2.1.1 Basic Information Coverage
2.1.1.1 Line item information completeness
2.1.1.2 Product description appropriateness
2.1.2 Coverage of other Contextual Information
2.1.2.1 …………………………………………………………

26. Measurable Attributes
• Attribute name
• Description and purpose Once you have a
model (what you
• How to measure
are going to
• What is measured measure), then
• Measurement/Calculation you start doing IT!
• Range (min, max)
• Objective
• Current

27. Usability Measurement
Attribute Scale How Measure or Objective Current
Calculation
Help Percent of Compare % 90% 40%
completeness Menu items menus and
with help items
help
Ease of Keystrokes Sample 50 % 3 12
access to items
find/use a
feature/func
tion/informa
tion
Consistency Number Examine integer 1 5
locations for menus and
same button doc.
Accuracy Number Collect from Integer or % <5 10
reported log files
errors
27

28. Usability Measurement Methods
Focus Walk
groups
Throughs
Satisfaction
Surveys
Labs
Heuristic
Evaluation
Logging
Let’s get started

29. Current Research-Summary of Usability
Evaluation Methods

30. Usability Logging
Measurement and Data Collection
• Identify users by using session
ID to identify a unique user.
• Iteratively insert code into the
application
• Collect data
• Analyze the data for each
attribute in different
dimensions and aggregations
• Determine the need for further
calculations and what attributes
to measure further
• Revise the data we are
collecting, adding or decreasing
granularity

31. Satisfaction Surveys
The process:
1. Calculate the usability score (satisfaction) of version X
2. Do the survey
3. Change to version X.1 – make changes to the
software directly correlated to the usability factors to
either increase or decrease the usability score
4. Do the survey again
5. See if differences made change impact the survey
results

32. Notes on Satisfaction and Usability
don’t have
what I want I’m unsatisfied • Satisfaction is a
subjective feeling
Highly usable dependent on many
software things other than
usability:
My password
doesn’t work
– A user can be highly
satisfied but the
application with low
Nice weather I’m usability.
today satisfied! – An application can be
! highly usable (high
low usability
software usability) but the user is
not satisfied!
Finished
my work today

33. Sample Evaluation
External Quality Requirements Measure EI value P/GI value
Global Quality Indicator 61.97%
1 Usability 60.88%
1.1 Understandability 83%
1.1.1 Icon/label ease to be recognized 100%
1.1.2 Information grouping cohesiveness 66%
1.2 Learnability 51.97%
1.2.1 ……………………………………………… …
1.3 Operability 49.50%
1.3.1 Control permanence 100%
1.3.2 Expected behaviour 50%
2 Content Quality 63.05%
2.1 Content Suitability 63.05%
2.1.1 Basic Information Coverage 50%
2.1.1.1 Line item information completeness 2 50%
2.1.1.2 Product description appropriateness 50%
2.1.2 Coverage of other Contextual Information 76.89%
2.1.2.1 ……………………………………………….. …
2.1.2.2 Return policy information completeness 33%

34. Healthcare Software
Heuristic Evaluation

35. Conclusion
• Usability is an abstract concept
• Defining usability is different for each
organization
• Need a model for your organization
• The model is the foundation of what to
measure
• Once you can measure, then you can evaluate
and improve

36. Next Steps
• Produce an action plan
– What usability attributes are important to your
organization?
• Develop a model
– What data can you collect/Which technique can
you use
• Maybe some elements of the model drop out-can’t be
measured that easily
– Start collecting and developing benchmarks
• Discuss with your manager/team

37. Thanks
Questions and Answers