Uid formative evaluation


Published on

Published in: Education, Technology, Design
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Uid formative evaluation

  1. 1. BS3001 Human Computer Interaction Usability Evaluation formative techniques By Jenny Le Peuple 2007 Edited by Dr K. Dudman 2008 Revised by Pen Lister 2009
  2. 2. Usability Evaluation <ul><li>“ Any object, product, system or service that will be used by humans has the potential for usability problems and should be subjected to some form of usability engineering ” </li></ul><ul><li>Nielsen, 1993 </li></ul>
  3. 3. System acceptability revisited... System acceptability Social acceptability Practical acceptability Efficient to use Cost Compatibility Reliability Etc Utility Usefulness From Nielsen (1993 p.25) Usability Easy to learn Easy to remember Few errors Subjectively pleasing
  4. 4. Usability evaluation OR Functionality testing <ul><li>Testing, objective: </li></ul><ul><ul><li>test functionality of system </li></ul></ul><ul><ul><ul><li>identify & fix bugs, faulty logic etc </li></ul></ul></ul><ul><li>Evaluation, objective: </li></ul><ul><ul><li>test usability of system </li></ul></ul><ul><ul><ul><li>can users achieve their goals in terms of: </li></ul></ul></ul><ul><ul><ul><ul><li>effectiveness </li></ul></ul></ul></ul><ul><ul><ul><ul><li>efficiency </li></ul></ul></ul></ul><ul><ul><ul><ul><li>productivity </li></ul></ul></ul></ul><ul><ul><ul><ul><li>safety </li></ul></ul></ul></ul><ul><ul><ul><ul><li>user satisfaction </li></ul></ul></ul></ul><ul><ul><ul><ul><li>... </li></ul></ul></ul></ul>
  5. 5. Usability problems <ul><li>Functionality testing is clearly important; </li></ul><ul><li>we focus on usability evaluation in this module </li></ul><ul><li>Many examples of poor interface design </li></ul><ul><li>Many products (not just IT-based) released with apparently little or no usability evaluation conducted </li></ul>
  6. 6. Usability problems - some examples Examples are from a flickr.com group about bad usability: http://www.flickr.com/groups/everyday-usability Where do I click to select the pump I want to use?
  7. 7. Usability problems - some examples Whaaa? These are the lift summoning controls at the Barbican
  8. 8. Usability problems - some examples This is the page you arrive at when you click on ‘special offers’ – hands up what is wrong with that….
  9. 9. Usability problems - some examples This is from MS Outlook – a great example of bad usability as a result of lack of consistency in design of the interface
  10. 10. Usability problems - some examples ABOVE: The worst possible way of entering numbers LEFT: The checkbox button should be radio buttons AND really should be either/or, don’t you think? This example is from Iarchitect “Hall of Shame”. It is archived at http://homepage.mac.com/bradster/iarchitect/
  11. 11. Usability evaluation –purpose <ul><li>Goals: </li></ul><ul><ul><li>assess level of effectiveness </li></ul></ul><ul><ul><li>assess effect of interface on users </li></ul></ul><ul><ul><li>identify specific problems </li></ul></ul><ul><li>Can inform user requirements elicitation </li></ul><ul><li>Obtain a measure of the extent to which the design meets original usability goals </li></ul><ul><ul><li>changes can be made to optimise the design </li></ul></ul><ul><ul><li>(as early as possible) </li></ul></ul>
  12. 12. General usability goals <ul><li>ISO 9241 </li></ul><ul><ul><li>efficiency </li></ul></ul><ul><ul><li>effectiveness </li></ul></ul><ul><ul><li>satisfaction </li></ul></ul><ul><li>Nielsen (1993) </li></ul><ul><ul><li>efficient </li></ul></ul><ul><ul><li>easy to learn </li></ul></ul><ul><ul><li>easy to remember </li></ul></ul><ul><ul><li>few errors </li></ul></ul><ul><ul><li>subjectively pleasing </li></ul></ul>‘ Usability Engineering’, Nielsen, 1993, p 26
  13. 13. The five “Es” review <ul><li>Quesenbery (2001) expanded ISO 9241 guidelines to five dimensions </li></ul><ul><ul><li>Effective </li></ul></ul><ul><ul><ul><li>completeness and accuracy with which users achieve specified goals </li></ul></ul></ul><ul><ul><li>Efficient </li></ul></ul><ul><ul><ul><li>the speed (with accuracy) in which users can complete the tasks for which they use the product </li></ul></ul></ul><ul><ul><li>Engaging </li></ul></ul><ul><ul><ul><li>pleasant and satisfying to use </li></ul></ul></ul><ul><ul><li>Error tolerant </li></ul></ul><ul><ul><ul><li>prevent errors caused by the user’s interaction & help the user recover from any errors that do occur </li></ul></ul></ul><ul><ul><li>Easy to learn </li></ul></ul><ul><ul><ul><li>allows users to build on their knowledge without deliberate effort </li></ul></ul></ul>Quesenbery, W. (2003). “The five dimensions of usability” in M. J. Albers & Mazur, B. (Eds) Content and Complexity . Lawrence Erlbaum.
  14. 14. Refining general guidelines into usability goals an example for a specific system (conference registration site) adapted from Quesenbery, 2003 You may find it useful, for your coursework, to group your usability goals under the same dimensions. Obviously there should be a lot more of them, since the above is just an example Dimension (guideline) Usability goal Effective Fewer than 5% of registrations will have errors that require follow-up by conference staff Efficient User will successfully complete registration in < 5 minutes Engaging In a follow-up survey, at least 80% of users will express comfort with using the online system instead of a paper system Error tolerant System will validate accommodation, meal and tutorial choices and allow user to confirm Easy to learn 95% of users will be able to successfully complete registration at first attempt
  15. 15. <ul><li>Identify task(s) to be tested </li></ul><ul><ul><li>use scenarios </li></ul></ul><ul><li>Select appropriate technique </li></ul><ul><li>Recruit typical users of the product </li></ul><ul><ul><li>(and/or expert) </li></ul></ul><ul><li>Conduct evaluation </li></ul><ul><li>Analyse results </li></ul><ul><li>Interpret the results </li></ul><ul><li>Make changes to design as necessary </li></ul>Evaluation methods - A generic framework
  16. 16. E valuation - techniques <ul><li>Large number of tools available </li></ul><ul><li>Various classification schemes </li></ul><ul><ul><li>Karat (1998) </li></ul></ul><ul><ul><li>Nielsen (1993) </li></ul></ul><ul><ul><li>see Christie et al (1995) for an overview </li></ul></ul><ul><li>Some techniques are multipurpose Prototyping can be used: </li></ul><ul><ul><li>to elicit requirements; </li></ul></ul><ul><ul><li>for design purposes </li></ul></ul><ul><ul><li>for evaluation (sometimes simultaneously) </li></ul></ul>
  17. 17. Numerous techniques available Proactive Field Study Pluralistic Walkthroughs Teaching Method Shadowing Method Co-discovery Learning Question-asking Protocol Scenario-based Checklists Heuristic Evaluation Thinking-aloud Protocol Cognitive Walkthroughs Paper Prototyping Usability Audits Expert Evaluation Coaching Method Performance Measurement Interviews Retrospective Testing Remote Testing Feature Inspection Focus Groups Questionnaires Field Observation Logging Actual Use
  18. 18. Evaluation methods - approaches <ul><li>Hewitt (1986) makes distinction between: </li></ul><ul><ul><li>formative evaluation </li></ul></ul><ul><ul><li>& </li></ul></ul><ul><ul><li>summative evaluation </li></ul></ul>
  19. 19. Formative evaluation - characteristics <ul><li>Carried out early in design process </li></ul><ul><ul><li>changes cheaper, easier to implement </li></ul></ul><ul><li>Should be iterative </li></ul><ul><li>Provides informal, usually qualitative indications of usability </li></ul><ul><li>Often employs “ quick & dirty ” techniques </li></ul><ul><ul><li>relatively easy </li></ul></ul><ul><ul><li>relatively low cost </li></ul></ul><ul><ul><li>results relatively quick to analyse & interpret </li></ul></ul>
  20. 20. Summative evaluation - characteristics <ul><li>Carried out in final stages of design process </li></ul><ul><li>Usually provides objective (often quantitative) measures of usability, e.g. </li></ul><ul><ul><li>to compare different designs </li></ul></ul><ul><ul><li>for marketing purposes </li></ul></ul><ul><li>Generally employs more “scientific” techniques: </li></ul><ul><ul><li>expertise needed to apply </li></ul></ul><ul><ul><li>can be costly </li></ul></ul><ul><ul><li>can be complex to analyse </li></ul></ul><ul><li>By now, too late to make major changes </li></ul>
  21. 21. Developing an evaluation plan <ul><li>Need to relate technique/tool to stage of product development </li></ul><ul><li>Key issues: </li></ul><ul><ul><li>setting usability goals (otherwise there is nothing to measure against) </li></ul></ul><ul><ul><li>selecting technique(s) </li></ul></ul><ul><ul><li>selecting members of evaluation team </li></ul></ul><ul><ul><li>logistical issues etc... </li></ul></ul>
  22. 22. Issues <ul><li>Usability goals </li></ul><ul><ul><li>need to be specific & measurable: WHAT ARE YOU TESTING? </li></ul></ul><ul><li>Techniques </li></ul><ul><ul><li>multiple? (convergent validity) WHICH METHODS WILL YOU USE? HOW MANY, CAN YOU COMPARE THEM MEANINGFULLY? </li></ul></ul><ul><li>Evaluation team </li></ul><ul><ul><li>interface designers </li></ul></ul><ul><ul><li>users </li></ul></ul><ul><ul><li>UI specialists/experts </li></ul></ul><ul><li>Logistical </li></ul><ul><ul><li>resources </li></ul></ul><ul><ul><ul><li>equipment </li></ul></ul></ul><ul><ul><ul><li>subjects </li></ul></ul></ul><ul><ul><ul><li>accommodation etc. </li></ul></ul></ul><ul><ul><li>timescales </li></ul></ul><ul><ul><li>ethical considerations </li></ul></ul>
  23. 23. Quantitative & qualitative data <ul><li>Quantitative </li></ul><ul><ul><li>Relatively objective e.g. </li></ul></ul><ul><ul><ul><li>how many mistakes have been made </li></ul></ul></ul><ul><ul><ul><li>how long to complete task </li></ul></ul></ul><ul><li>Qualitative </li></ul><ul><ul><li>Relatively subjective e.g. </li></ul></ul><ul><ul><ul><li>user’s attitude to software </li></ul></ul></ul><ul><li>think about which techniques you could use to obtain measurements of the above </li></ul>
  24. 24. Writing up an evaluation report <ul><li>Whatever techniques are used: </li></ul><ul><ul><li>type of technique should be explained (& why chosen) </li></ul></ul><ul><ul><li>include a description of the process e.g. </li></ul></ul><ul><ul><ul><li>why </li></ul></ul></ul><ul><ul><ul><li>who </li></ul></ul></ul><ul><ul><ul><li>where </li></ul></ul></ul><ul><ul><ul><li>when etc. </li></ul></ul></ul><ul><ul><li>raw data should be: </li></ul></ul><ul><ul><ul><li>recorded accurately </li></ul></ul></ul><ul><ul><ul><li>summarised in report (e.g. in a table) </li></ul></ul></ul><ul><ul><li>data can then be: </li></ul></ul><ul><ul><ul><li>analysed </li></ul></ul></ul><ul><ul><ul><li>interpreted: what do the results mean </li></ul></ul></ul><ul><ul><ul><li>(the hard part – on their own, numbers mean little) </li></ul></ul></ul><ul><ul><li>what changes need to be made in the light of your findings? </li></ul></ul>
  25. 25. Writing up an evaluation report <ul><li>How to write the report: </li></ul><ul><ul><li>Introduction </li></ul></ul><ul><ul><li>Tasks to be tested </li></ul></ul><ul><ul><li>User Group(s) </li></ul></ul><ul><ul><li>Methods </li></ul></ul><ul><ul><li>Analysis and Results </li></ul></ul><ul><ul><li>Discussion </li></ul></ul><ul><ul><li>Conclusion </li></ul></ul><ul><ul><li>Appendices: ALL raw data, example questionnaires, interview </li></ul></ul><ul><ul><li>scripts, copies of paper prototypes (if used), audio & video files if made etc </li></ul></ul>* Remember, when carrying out research of this type, it is important to make a statement about privacy, confidentiality and anonymity of participants. This would be a legal requirement of any ‘real’ research.
  26. 26. Conclusion: <ul><li>Any usability evaluation is better than none </li></ul><ul><li>Each iteration will reveal more flaws </li></ul><ul><li>Need not involve end users at all stages </li></ul><ul><li>e.g. expert appraisal can give useful findings </li></ul><ul><li>Need not be expensive or take a lot of time e.g. Nielsen's “Discount Usability Engineering” (‘Usability Engineering’, Nielsen, 1993, p 17; “Lost our lease Usability testing”, in ‘Don’t Make Me Think, Steve Krug, 2000, p143-144) </li></ul><ul><li>Many different methods available </li></ul><ul><li>do some research for your projects </li></ul><ul><li>e.g. paper prototyping is a type of formative evaluation </li></ul>
  27. 27. Activities <ul><li>Read: Chapter 6 “Usability Evaluation: formative techniques” in Le Peuple, J. & Scane, R. (2003). User Interface Design . Crucial. </li></ul><ul><li>Visit: </li></ul><ul><li>UPA (Usability Professionals Association) http://www.upassoc.org/upa_publications/jus/2005_november/formative.html </li></ul><ul><li>and download/ read the article “ Towards the Design of Effective Formative Test Reports” (it may be useful for your coursework). </li></ul><ul><li>UserDesign </li></ul><ul><li>http://www.userdesign.com/usability_uem.html </li></ul><ul><li>and read the comparisons between different evaluation methods. </li></ul><ul><li>James Hom’s Usability Methods Toolbox </li></ul><ul><li>http://jthom.best.vwh.net/usability/ </li></ul><ul><li>WQ Usability (Whitney Quesenbery’s company website) </li></ul><ul><li>http://www.wqusability.com/articles/getting-started.html </li></ul><ul><li>lots of useful material to download and read, especially relating to the “5 Es”) </li></ul>
  28. 28. References/further reading #1 <ul><li>Beyer, H. & Holtzblatt (1997) Contextual Design : A Customer-Centered Approach to Systems Designs. Morgan Kaufmann Publishers . </li></ul><ul><li>Christie, B., Scane, R. & Collyer, J. (1995) “Evaluation of human-computer interaction at the user interface to advanced IT systems” in J.R. Wilson,. & N. Corlett, Eds. Evaluation of Human work: A Practical Ergonomics Methodology (2nd edition). Taylor and Francis, </li></ul><ul><li>Dix, A. et al (1998) Human-Computer Interaction (2nd Edition). Prentice Hall Europe. </li></ul><ul><li>Faulkner, X. (2000) Usability Engineering, Macmillan Press Ltd. </li></ul><ul><li>Forsythe, C. Grose, E. & Ratner, J., Eds. (1998) Human Factors and Web Development . Lawrence Erlbaum Associates. </li></ul><ul><li>Hewitt, T (1986) “Iterative evaluation” in M.D. Harrison & A.F. Monks, (Eds.) People and Computers: Designing for Usability , Proceedings of the Second Conference of the BCS HCI Specialist Group. </li></ul><ul><li>Karat, J. (1988) “Software Evaluation Methodologies” in M. Helander, Ed. Handbook of Human-Computer Interaction . Elsevier Science Publishers. </li></ul>
  29. 29. References/further reading #2 <ul><li>Le Peuple, J. & Scane, R. (2003). User Interface Design. Crucial. </li></ul><ul><li>Mayhew, D.J. (1999) The Usability Engineering Lifecycle: A Practitioner's Handbook for User Interface Design . Morgan Kaufmann </li></ul><ul><li>Nielsen, J. (1993) Usability Engineering . AP Professional. </li></ul><ul><li>Nielsen, J & Mack, R. L., Eds. (1994) Usability Inspection Methods. John Wiley & Sons, Inc. </li></ul><ul><li>Norman, D.A. (1988) The Design of Everyday Things, Basic Books. (First published as The Psychology of Everyday Things ). </li></ul><ul><li>Preece, J. et al (1995) Human-Computer Interaction. Addison-Wesley. </li></ul><ul><li>Quesenbery, W. (2003). “The five dimensions of usability” in M. J. Albers & Mazur, B. (Eds) Content and Complexity . Lawrence Erlbaum. </li></ul><ul><li>Rubin, J. (1994) Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. John Wiley & Sons </li></ul>
  30. 30. Useful Websites http://www.worldusabilityday.org/ http://www.useit.com http://www.uie.com/brainsparks http://www.usability.gov / http://www.usabilitynet.org/home.htm Published in Interactions September + October 2005
  31. 31. End