Streamlined Product Evaluation


Published on

When you have limited resources and no access to users, this process will allow you to evaluate a product's usability and present findings and recommendations to the product team.

Published in: Economy & Finance, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Streamlined Product Evaluation

    1. 1.
    2. 2. Catherine Weeks SeaChange International
    3. 3. &  Product Evaluation
    4. 4. <ul><li>Expert Review </li></ul><ul><li>Heuristic Review </li></ul><ul><li>Cognitive Walkthrough </li></ul>“ This is a more specific version of a design walkthrough, focusing on cognitive principles . Based on a user's goals, a group of evaluators steps through tasks, evaluating at each step how difficult it is for the user to identify and operate the interface element most relevant to their current subgoal and how clearly the system provides feedback to that action. Cognitive walkthroughs take into consideration the user's thought processes that contribute to decision making, such as memory load and ability to reason .” – Usability First ( “ Heuristic evaluations usually are conducted by a small set (one to three) of evaluators . The evaluators independently examine a user interface and judge its compliance with a set of usability principles . The result of this analysis is a list of potential usability issues or problems. The usability principles, also referred to as usability heuristics, are taken from published lists . Ideally, each potential usability problem is assigned to one or more heuristics to help facilitate fixing the problem. As more evaluators are involved, more true problems are found.” – ( “ Expert usability reviews are a fast and economical route to improving the usability of a Web site... The report outlines the criteria used to evaluate the site with reliable recommendations, which are based on extensive Human-Computer Interaction (HCI) research and experience .” – Mich. State Usability & Accessibility Center (
    5. 5. <ul><li>Expert Review </li></ul><ul><ul><li>Experience </li></ul></ul><ul><ul><li>Research </li></ul></ul><ul><ul><li>Many evaluators </li></ul></ul><ul><li>Heuristic Review </li></ul><ul><ul><li>Principles </li></ul></ul><ul><ul><li>Non-expert </li></ul></ul><ul><ul><li>Comparable </li></ul></ul><ul><li>Cognitive Walkthrough </li></ul><ul><ul><li>Task centric </li></ul></ul><ul><ul><li>Working prototype </li></ul></ul><ul><ul><li>Timely </li></ul></ul>
    6. 6. <ul><li>Limited or no product team access </li></ul><ul><li>Limited UX resources </li></ul><ul><li>Limited time </li></ul><ul><li>No access to users </li></ul><ul><li>Access to working applications </li></ul><ul><li>This led me to create a process based on a twist of the product evaluation types… </li></ul>
    7. 7. <ul><li>Identify User </li></ul><ul><li>Product Review </li></ul><ul><ul><li>Gives 1,000 foot view </li></ul></ul><ul><ul><li>Looks at branding, product placement, end user satisfaction </li></ul></ul><ul><li>Cognitive Walkthrough </li></ul><ul><ul><li>Look through application based on user tasks </li></ul></ul><ul><ul><li>Find interaction problems, layout issues, and even bugs! </li></ul></ul><ul><li>Create Recommendations </li></ul><ul><li>Produce Report </li></ul>
    8. 8. <ul><li>Case Intro: </li></ul><ul><li>No team </li></ul><ul><li>&quot;Pet&quot; project </li></ul><ul><li>No requirements, no business case </li></ul><ul><li>Natural extension to current products </li></ul><ul><li>Identify User: </li></ul><ul><li>Technical </li></ul><ul><li>Business solution </li></ul>
    9. 9. <ul><li>Product Review Findings </li></ul><ul><li>“ Customers will not be able to use [Product name] outside of [Parent product name], therefore creating a barrier to use…” </li></ul><ul><li>“… providers could alter the recommendations without any indication to the user that these are ‘sponsored’ rather than purely calculated recommendations.” </li></ul><ul><li>Cognitive Walkthrough Findings </li></ul><ul><li>If the user selects the incorrect operation, in order to fix their mistake, the user must delete the new rule, and add another rule… </li></ul><ul><li>The colors do not support color blindness (red and green used) </li></ul>
    10. 10. <ul><li>Case Intro: </li></ul><ul><li>Offshore office (Asia) </li></ul><ul><li>Customer buy-in </li></ul><ul><li>Responsive team </li></ul><ul><li>Identify User: </li></ul><ul><li>Technical </li></ul><ul><li>Business solution </li></ul>
    11. 11. <ul><li>Product Review Findings </li></ul><ul><li>It is unclear where the user should begin in order to make the system work, and would require a vast amount of training in order for the users to become proficient in the system. </li></ul><ul><li>Correct SeaChange product branding was not used. </li></ul><ul><li>Cognitive Walkthrough Findings </li></ul><ul><li>It is unclear how to access this application when receiving the hardware. </li></ul><ul><li>The content area pane is laid out well and supports the user’s work. The whole page or continuing page look helps users identify if they need to scroll to see more information. </li></ul>
    12. 12. <ul><li>Requires few resources </li></ul><ul><li>Can be as minimal or extensive as needed </li></ul><ul><li>Produces a report that is easy for the product team to work with </li></ul>
    13. 13. <ul><li>Relies on experience and research, not user input! </li></ul><ul><li>Reliance on few people to catch many issues </li></ul><ul><li>Requires understanding of business needs </li></ul><ul><li>Limited team collaboration opportunities </li></ul>
    14. 14. <ul><li>Access to a working interface or prototype </li></ul><ul><li>Product Review: 1 ~ 3 days </li></ul><ul><li>Cognitive Walkthrough: 3 ~ 5 days </li></ul><ul><li>Recommendations: 2 ~ 5 days </li></ul><ul><li>Report: 2 ~ 4 days </li></ul><ul><li>Total: 2 ~ 4 weeks </li></ul>
    15. 15. <ul><li>Product is perceived as valuable to company strategy </li></ul><ul><li>Limited resources </li></ul><ul><li>No access to users </li></ul>
    16. 16. <ul><li>Works well in limited resource situations </li></ul><ul><li>Can provide minimal to extensive reviews </li></ul><ul><li>Should always opt to have users involved if possible </li></ul><ul><li>Results in a clear report for the product team </li></ul>
    17. 17. <ul><li>Product Review outline </li></ul><ul><li>Example of Cognitive Walkthrough </li></ul><ul><li>Report layout example </li></ul><ul><li>[email_address] </li></ul>
    18. 18. <ul><li>OK/Cancel Comics: </li></ul><ul><li> , </li></ul><ul><li>Cognitive Walkthrough definition: </li></ul><ul><li>Heuristic Review definition: </li></ul><ul><li> </li></ul><ul><li>Expert Review definition: </li></ul><ul><li> </li></ul>