Meta-Analysis of Correlations Among Usability Measures Kasper Hornbæk & Effie Law University of Copenhagen & Eidgenössisch...
Background and aim <ul><li>Usability describes quality-in-use  </li></ul><ul><li>Usability may be measured in many ways, e...
Earlier work <ul><li>Many classifications of usability (e.g., ISO 1998, Seffah et al. 2006, Hornbæk 2006) </li></ul><ul><l...
Data collection <ul><li>Studies from 8 HCI journals and conferences </li></ul><ul><li>Consider as candidates original rese...
Analysis <ul><li>Measures in each study classified according to  </li></ul><ul><ul><li>ISO 9241-11: effectiveness, efficie...
Results:  which measures are used? <ul><li>Effectiveness is typically error rates or task completion </li></ul>
Results:  which measures are used? <ul><li>Efficiency is typically measured as time </li></ul>
Results:  which measures are used? <ul><li>Satisfaction is measured in many ways, but standard questionnaires are rarely u...
Results: are measures correlated? <ul><li>Effectiveness vs. efficiency: r = .247 ± .059 </li></ul><ul><li>May be interpret...
Results: are measures correlated? <ul><li>Task complexity does  not influence the  correlations </li></ul><ul><li>More com...
Results: are measures correlated? <ul><li>Effectiveness vs. satisfaction: r = .164 ± .062 </li></ul><ul><li>Preference is ...
Results: are measures correlated? <ul><li>Efficiency vs. satisfaction: r = .196 ± .064 </li></ul><ul><li>Interfaces that a...
Results: other findings <ul><li>We calculated the reliability of questionnaires </li></ul><ul><li>Homegrown questionnaires...
Discussion <ul><li>Two interpretations of the main result </li></ul><ul><ul><li>The half-empty interpretation: correlation...
Conclusion <ul><li>We find mostly small correlations among usability measures </li></ul><ul><li>These correlations are sha...
Upcoming SlideShare
Loading in …5
×

CHI2007: Meta-analysis of correlations among usability measurs

1,389 views

Published on

A presentation of a paper at CHI2007 on

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,389
On SlideShare
0
From Embeds
0
Number of Embeds
44
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • CHI2007: Meta-analysis of correlations among usability measurs

    1. 1. Meta-Analysis of Correlations Among Usability Measures Kasper Hornbæk & Effie Law University of Copenhagen & Eidgenössische Technische Hochschule Zürich
    2. 2. Background and aim <ul><li>Usability describes quality-in-use </li></ul><ul><li>Usability may be measured in many ways, e.g., as task completion time, error rates, subjective satisfaction, perceived workload, etc. </li></ul><ul><li>Quantitative measures of usability play an important role in human-computer interaction </li></ul><ul><li>Our aim is to understand how usability measures relate, which will help to: </li></ul><ul><ul><li>Select measures for usability studies </li></ul></ul><ul><ul><li>Understand usability better </li></ul></ul>
    3. 3. Earlier work <ul><li>Many classifications of usability (e.g., ISO 1998, Seffah et al. 2006, Hornbæk 2006) </li></ul><ul><li>Studies of correlations among measures (e.g., Nielsen & Levy 1994, Sauro & Kindlund 2005) </li></ul><ul><li>Limitations of existing studies include: </li></ul><ul><ul><li>Contradictory findings (e.g., Nielsen & Levy 1994 vs. Frøkjær et al. 2000) </li></ul></ul><ul><ul><li>Mostly without access to raw data (e.g., Nielsen & Levy 1994) </li></ul></ul><ul><ul><li>Do not consider the variety of ways usability may be measured or moderator variables </li></ul></ul>
    4. 4. Data collection <ul><li>Studies from 8 HCI journals and conferences </li></ul><ul><li>Consider as candidates original research papers reporting usability measures </li></ul><ul><li>Contacted authors of these papers and end up with raw data from 73 studies </li></ul>
    5. 5. Analysis <ul><li>Measures in each study classified according to </li></ul><ul><ul><li>ISO 9241-11: effectiveness, efficiency, satisfaction </li></ul></ul><ul><ul><li>Hornbæk (2006)’s 54 types of measure </li></ul></ul><ul><li>Then correlations among types of usability measure are calculated </li></ul><ul><li>These correlations may then be integrated using techniques from meta-analysis (Rosenthal 1991) </li></ul>
    6. 6. Results: which measures are used? <ul><li>Effectiveness is typically error rates or task completion </li></ul>
    7. 7. Results: which measures are used? <ul><li>Efficiency is typically measured as time </li></ul>
    8. 8. Results: which measures are used? <ul><li>Satisfaction is measured in many ways, but standard questionnaires are rarely used </li></ul>
    9. 9. Results: are measures correlated? <ul><li>Effectiveness vs. efficiency: r = .247 ± .059 </li></ul><ul><li>May be interpreted as: </li></ul><ul><ul><li>6% variance explained </li></ul></ul><ul><ul><li>small (~.1) to medium (~.3) effect (Cohen 1969) </li></ul></ul><ul><ul><li>examples: </li></ul></ul>r = .229 r = .23
    10. 10. Results: are measures correlated? <ul><li>Task complexity does not influence the correlations </li></ul><ul><li>More complex measures (e.g., quality of outcome) attenuate correlations </li></ul><ul><li>Difference between errors-along-the-way (.441) and task-completion-errors (.155) </li></ul>
    11. 11. Results: are measures correlated? <ul><li>Effectiveness vs. satisfaction: r = .164 ± .062 </li></ul><ul><li>Preference is related to fewer errors </li></ul><ul><ul><li>Prefer (13% errors) vs. do not prefer (18% errors) </li></ul></ul><ul><li>Six studies measure both effectiveness (objective) and participants’ assessment of their effectiveness (subjective) </li></ul><ul><ul><li>“ were your answers to tasks: very good – very poor” vs. errors in task answers </li></ul></ul><ul><ul><li>Correlations of these measures are not significantly different from zero </li></ul></ul>
    12. 12. Results: are measures correlated? <ul><li>Efficiency vs. satisfaction: r = .196 ± .064 </li></ul><ul><li>Interfaces that are preferred are about 20% faster than non-preferred ones </li></ul><ul><li>Again no correlation between objective and subjective measures </li></ul>
    13. 13. Results: other findings <ul><li>We calculated the reliability of questionnaires </li></ul><ul><li>Homegrown questionnaires have low reliability, 6 are below .7 </li></ul>
    14. 14. Discussion <ul><li>Two interpretations of the main result </li></ul><ul><ul><li>The half-empty interpretation: correlations are small ->can not reduce usability to one measure (e.g., Sauro & Kindlund 2005) </li></ul></ul><ul><ul><li>The half-full interpretation: surprising with consistent correlations across very different studies </li></ul></ul><ul><li>Models of usability may be improved </li></ul><ul><ul><li>Subjective/objective, error types not separated in some usability models </li></ul></ul><ul><li>Recommendations for practice </li></ul>
    15. 15. Conclusion <ul><li>We find mostly small correlations among usability measures </li></ul><ul><li>These correlations are shaped by measure complexity and the use of subjective/objective measures, but not by task complexity </li></ul><ul><li>Suggests that models of usability need extension and that usability measures should not be combined into a single measures </li></ul>kash@diku.dk [email_address]

    ×