Meta-Analysis of Correlations Among Usability Measures Kasper Hornbæk & Effie Law University of Copenhagen & Eidgenössische Technische Hochschule Zürich

Background and aim Usability describes quality-in-use Usability may be measured in many ways, e.g., as task completion time, error rates, subjective satisfaction, perceived workload, etc. Quantitative measures of usability play an important role in human-computer interaction Our aim is to understand how usability measures relate, which will help to: Select measures for usability studies Understand usability better

Usability describes quality-in-use

Usability may be measured in many ways, e.g., as task completion time, error rates, subjective satisfaction, perceived workload, etc.

Quantitative measures of usability play an important role in human-computer interaction

Our aim is to understand how usability measures relate, which will help to:

Select measures for usability studies

Understand usability better

Earlier work Many classifications of usability (e.g., ISO 1998, Seffah et al. 2006, Hornbæk 2006) Studies of correlations among measures (e.g., Nielsen & Levy 1994, Sauro & Kindlund 2005) Limitations of existing studies include: Contradictory findings (e.g., Nielsen & Levy 1994 vs. Frøkjær et al. 2000) Mostly without access to raw data (e.g., Nielsen & Levy 1994) Do not consider the variety of ways usability may be measured or moderator variables

Many classifications of usability (e.g., ISO 1998, Seffah et al. 2006, Hornbæk 2006)

Studies of correlations among measures (e.g., Nielsen & Levy 1994, Sauro & Kindlund 2005)

Limitations of existing studies include:

Contradictory findings (e.g., Nielsen & Levy 1994 vs. Frøkjær et al. 2000)

Mostly without access to raw data (e.g., Nielsen & Levy 1994)

Do not consider the variety of ways usability may be measured or moderator variables

Data collection Studies from 8 HCI journals and conferences Consider as candidates original research papers reporting usability measures Contacted authors of these papers and end up with raw data from 73 studies

Studies from 8 HCI journals and conferences

Consider as candidates original research papers reporting usability measures

Contacted authors of these papers and end up with raw data from 73 studies

Analysis Measures in each study classified according to ISO 9241-11: effectiveness, efficiency, satisfaction Hornbæk (2006)’s 54 types of measure Then correlations among types of usability measure are calculated These correlations may then be integrated using techniques from meta-analysis (Rosenthal 1991)

Measures in each study classified according to

ISO 9241-11: effectiveness, efficiency, satisfaction

Hornbæk (2006)’s 54 types of measure

Then correlations among types of usability measure are calculated

These correlations may then be integrated using techniques from meta-analysis (Rosenthal 1991)

Results: which measures are used? Effectiveness is typically error rates or task completion

Effectiveness is typically error rates or task completion

Results: which measures are used? Efficiency is typically measured as time

Efficiency is typically measured as time

Results: which measures are used? Satisfaction is measured in many ways, but standard questionnaires are rarely used

Satisfaction is measured in many ways, but standard questionnaires are rarely used

Results: are measures correlated? Effectiveness vs. efficiency: r = .247 ± .059 May be interpreted as: 6% variance explained small (~.1) to medium (~.3) effect (Cohen 1969) examples: r = .229 r = .23

Effectiveness vs. efficiency: r = .247 ± .059

May be interpreted as:

6% variance explained

small (~.1) to medium (~.3) effect (Cohen 1969)

examples:

Results: are measures correlated? Task complexity does not influence the correlations More complex measures (e.g., quality of outcome) attenuate correlations Difference between errors-along-the-way (.441) and task-completion-errors (.155)

Task complexity does not influence the correlations

More complex measures (e.g., quality of outcome) attenuate correlations

Difference between errors-along-the-way (.441) and task-completion-errors (.155)

Results: are measures correlated? Effectiveness vs. satisfaction: r = .164 ± .062 Preference is related to fewer errors Prefer (13% errors) vs. do not prefer (18% errors) Six studies measure both effectiveness (objective) and participants’ assessment of their effectiveness (subjective) “ were your answers to tasks: very good – very poor” vs. errors in task answers Correlations of these measures are not significantly different from zero

Effectiveness vs. satisfaction: r = .164 ± .062

Preference is related to fewer errors

Prefer (13% errors) vs. do not prefer (18% errors)

Six studies measure both effectiveness (objective) and participants’ assessment of their effectiveness (subjective)

“ were your answers to tasks: very good – very poor” vs. errors in task answers

Correlations of these measures are not significantly different from zero

Results: are measures correlated? Efficiency vs. satisfaction: r = .196 ± .064 Interfaces that are preferred are about 20% faster than non-preferred ones Again no correlation between objective and subjective measures

Efficiency vs. satisfaction: r = .196 ± .064

Interfaces that are preferred are about 20% faster than non-preferred ones

Again no correlation between objective and subjective measures

Results: other findings We calculated the reliability of questionnaires Homegrown questionnaires have low reliability, 6 are below .7

We calculated the reliability of questionnaires

Homegrown questionnaires have low reliability, 6 are below .7

Discussion Two interpretations of the main result The half-empty interpretation: correlations are small ->can not reduce usability to one measure (e.g., Sauro & Kindlund 2005) The half-full interpretation: surprising with consistent correlations across very different studies Models of usability may be improved Subjective/objective, error types not separated in some usability models Recommendations for practice

Two interpretations of the main result

The half-empty interpretation: correlations are small ->can not reduce usability to one measure (e.g., Sauro & Kindlund 2005)

The half-full interpretation: surprising with consistent correlations across very different studies

Models of usability may be improved

Subjective/objective, error types not separated in some usability models

Recommendations for practice

Conclusion We find mostly small correlations among usability measures These correlations are shaped by measure complexity and the use of subjective/objective measures, but not by task complexity Suggests that models of usability need extension and that usability measures should not be combined into a single measures kash@diku.dk [email_address]

We find mostly small correlations among usability measures

These correlations are shaped by measure complexity and the use of subjective/objective measures, but not by task complexity

Suggests that models of usability need extension and that usability measures should not be combined into a single measures