Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Paper Prototype Evaluation


Published on

Paper Prototyping @ EIA 2013

Published in: Education, Technology, Design
  • Be the first to comment

Paper Prototype Evaluation

  1. 1. Paper Prototype Evaluation
  2. 2. Evaluation • Well, before addressing the evaluation of paper prototypes, perhaps we should consider reasoning upon what we should evaluate, right? – Let’s then take a look into concepts such as usefulness, usability and user experience
  3. 3. Usefulness • Usefulness is a crucial quality of any product or service – Concerns the degree to which a product enables a user to achieve his or her goals, and is an assessment of the user’s willingness to use the product or service at all • Without usefulness, other measures make little or no sense, because it will just sit on the shelf • We name this the fundamental quality of a product or service
  4. 4. Usability • According to the ISO standard – Usability is the extent to which a product or service can be used by specific users to achieve predefined goals in a specified context of use – The standard defines three main usability dimensions: • Effectiveness, • Efficiency, and • Satisfaction
  5. 5. Usability • Anyway, when looking into usability, we normally account for: – Effectiveness – Efficiency – Satisfaction – Learnability – Memorability • We name these the pragmatic qualities of a product or service
  6. 6. Usability • Common reason for the delivery of less usable products are – System focused process – Poorly integrated teams – Design and development mismatches – And of course, designing usable products is difficult • That’s what we are pushing paper prototyping in this moment of the process
  7. 7. User Experience • According to its ISO standard, user experience is a person’s perception and responses that result from the use or anticipated use of a product or service
  8. 8. User Experience • User experience subsumes usability and includes includes – emotions, beliefs, preferences, perceptions, physic al and psychological responses, behaviors and accomplishments that occur before, during and after use • We name these the effected of the hedonic, or pleasure related, qualities of a product or service
  9. 9. Evaluation • As stating in the beginning, before addressing the evaluation of paper prototypes, we should consider reasoning upon what we should evaluate – And now that we briefly cover usefulness, usability and user experience, I believe it’s fair to say that we should target going as far as usability
  10. 10. Evaluation • In general, product and service evaluation can be formative or summative • Paper prototype evaluation is mostly formative
  11. 11. Evaluation • Paper prototype evaluation focus on… – The most significant issues preventing users from accomplishing their goals – What works and what do users find frustrating – What are the most common errors or mistakes users are making – Assessing the improvements being made from one design interaction to the next – Identifying issues that are expected to remain even when the product is launched
  12. 12. Evaluation • Today we look into – Inspection methods • These are methods where an expert evaluator inspects a product or service – Testing methods • These are methods where products and services are evaluated by testing them on real users
  13. 13. Inspection methods • There are also several but we will address: – Cognitive walk-through – Heuristic evaluation
  14. 14. Cognitive walk-through • The purpose is to verify if the paper prototype actually allows the fulfillment of the selected user stories – This is a within team activity that ensures that your prototype complies with the identified user stories
  15. 15. Cognitive walk-through • Designers and developers of the product or service then walk through the steps as a group, asking themselves a set of questions at each step – Data is gathered during the walk-through, and afterwards a report of potential issues is compiled – Finally the evaluated proposition is redesigned to address the issues identified
  16. 16. Heuristic evaluation • This is a type of evaluation ideally carried out by an expert. – It specifically involves evaluators examining the design and judging its compliance with recognized principles, such as Jakob Nielsen’s 10 usability heuristics • These evaluation methods are now widely taught and practiced in the new media sector, where products and services are often designed in a short space of time on a budget that may restrict the amount of money available to provide for other types of interface testing
  17. 17. Testing methods • There are several but we will address: – Co-discovery – Wizard of Oz
  18. 18. Co-discovery • Two users attempt to perform tasks together while being observed – They are to help each other in the same manner as they would if they were working together to accomplish a common goal using the product – They are encouraged to explain what they are thinking about while working on the tasks
  19. 19. Co-discovery • The designers and developers should refrain from explaining the design decisions and rather focus on getting the most of the pair of users tacking with your prototype – Note taking is fundamental and you should run this kind of test until no significant additional information is feed back into the design process
  20. 20. Wizard of OZ • This is a testing approach built upon a paper device instead of using a working technological artifact – This king of testing involves systematic observation under controlled conditions to determine how well people can use a product or service – Rather than showing users a rough draft and asking, Do you understand this?, this kind of testing involves watching people trying to use something for its intended purpose
  21. 21. Wizard of OZ • Setting up such a test involves asking the test subjects to recreate a set of user stories after being introduced to the underlying scenario – For example, to test the attachment function of an e-mail program, a scenario would describe a situation where a person needs to send an e-mail attachment, and ask him or her to undertake this task – The aim is to observe how people function in a realistic manner, so that developers can see problem areas, and what people like.
  22. 22. Testing methods • This kind of evaluation should be repeated until no significant added value comes from bring in an additional subject – The usual number is 5, but other claim otherwise
  23. 23. Pre and post-tests • When applying testing methods, pre-test and post-test questionnaires are also used to gather feedback on the product being tested – A common questionnaire is the 25 years old System Usability Scale
  24. 24. System Usability Scale
  25. 25. Evaluation protocol • But these methods are useless without an adequate evaluation protocol • While designing your evaluation protocol, you should take into account… – A mixed selection of inspection and testing methods; – The user stories supported by your prototype; and – The affordances of your paper prototype.
  26. 26. Evaluation protocol • Examples are provided in the companion blog but please note that… – In all cases, evaluation protocols should be piloted to ensure that once they are being applied, you are actually focusing in assessing the prototype and not on solving evaluation protocol issues
  27. 27. Now it’s up to you to make it happen
  28. 28. paperprototypingateia