Game Design 2 - Lecture 8 - Expert Evaluation


Published on

Lecture 8 in the class COMU346 at Caledonian University.

This lecture looks at Expert Evaluation techniques including Cognitive Walkthrough and Heuristic Evaluation

Published in: Technology, Design
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Game Design 2 - Lecture 8 - Expert Evaluation

  1. 1. [email_address] Game Design 2 Lecture 8: Expert Evaluation
  2. 2. Non End-User Evaluation <ul><li>Why? </li></ul><ul><li>Expert Evaluations </li></ul><ul><li>Design & Usability Heuristics </li></ul>
  3. 3. Expert Evaluations & Design / Usability Heuristics <ul><li>Will look at: </li></ul><ul><li>Need for alternatives to user evaluation </li></ul><ul><li>Methods of evaluating without end users (using expert evaluators) </li></ul><ul><li>Some heuristics / guidelines offered by experts </li></ul>
  4. 4. End User Evaluations <ul><li>End-user evaluations can be expensive </li></ul><ul><ul><li>The methods are very time consuming </li></ul></ul><ul><ul><li>Users may not be willing </li></ul></ul><ul><ul><li>To get truly ‘fresh’ eyes, so called “kleenex” testing requires different players each time </li></ul></ul><ul><li>Concerns about leaks </li></ul><ul><ul><li>Few external play testers at early stages </li></ul></ul><ul><ul><li>Friends & family play testers may be too kind </li></ul></ul>
  5. 5. Expert Evaluations <ul><li>As an alternative to some user testing, expert evaluators / testers can be used </li></ul><ul><li>Falconer details 10 inspection methods, we will look at two: </li></ul><ul><ul><li>Cognitive Walkthrough </li></ul></ul><ul><ul><li>Heuristic Evaluation </li></ul></ul>
  6. 6. Cognitive Walkthrough <ul><li>In this approach experts imitate users </li></ul><ul><ul><li>Relatively quick and cheap </li></ul></ul><ul><ul><li>Expert needs to be skilled and requires: </li></ul></ul><ul><ul><ul><li>A description of users (e.g. level of experience) </li></ul></ul></ul><ul><ul><ul><li>A description of system (or an operational system) </li></ul></ul></ul><ul><ul><ul><li>A description of the task to be carried out </li></ul></ul></ul><ul><ul><ul><li>A list of the actions required to complete the task </li></ul></ul></ul>
  7. 7. Cognitive Walkthrough <ul><li>Expert addresses questions such as: </li></ul><ul><ul><li>Is the goal clear at this stage? </li></ul></ul><ul><ul><li>Is the appropriate action obvious? </li></ul></ul><ul><ul><li>Is it clear that the appropriate action leads to the goal? </li></ul></ul><ul><ul><li>What problems (or potential problems) are there in performing the action? </li></ul></ul><ul><li>Essential that the expert tries to think like the end user and not like themselves. </li></ul>
  8. 8. Consider Dwarf Fortress <ul><li>A scenario: </li></ul><ul><ul><li>Goal is to build a farm plot </li></ul></ul><ul><ul><li>Is the appropriate action obvious? </li></ul></ul><ul><ul><ul><li>What if no dwarves are farmers? </li></ul></ul></ul><ul><ul><li>Is it clear that the appropriate action leads to the goal? </li></ul></ul><ul><ul><ul><li>Does the game help you? </li></ul></ul></ul><ul><ul><li>What problems (or potential problems) are there in performing the action? </li></ul></ul><ul><ul><ul><li>How do you know to give a dwarf ‘farming’ as a job? </li></ul></ul></ul><ul><li>I know how to do this but thinking as a ‘user’ it’s not so easy. </li></ul>
  9. 9. Heuristic Evaluation <ul><li>Involves assessing how closely an interface or system conforms to a predefined set of guidelines or heuristics. </li></ul><ul><li>Examples: </li></ul><ul><ul><li>Nielsen’s usability heuristics </li></ul></ul><ul><ul><li>Schneiderman’s eight golden rules </li></ul></ul><ul><ul><li>Norman’s seven principles </li></ul></ul>
  10. 10. Nielsen’s Usability Heuristics <ul><li>Give feedback </li></ul><ul><ul><li>keep users informed about what is happening </li></ul></ul><ul><li>Speak the user’s language </li></ul><ul><ul><li>dialogs should be expressed clearly using terms familiar to the user </li></ul></ul><ul><li>User control and freedom </li></ul><ul><ul><li>clearly marked exits and undo/redo </li></ul></ul><ul><li>Consistency and standards </li></ul><ul><li>Prevent errors </li></ul><ul><ul><li>even better than having good error messages </li></ul></ul>
  11. 11. Nielsen’s Usability Heuristics <ul><li>Minimise memory load </li></ul><ul><ul><li>recognition rather than recall </li></ul></ul><ul><li>Shortcuts </li></ul><ul><ul><li>accelerators (unseen by novices) speed up interactions for experts </li></ul></ul><ul><li>Aesthetic and minimalist design </li></ul><ul><ul><li>don’t have irrelevant or rarely needed information </li></ul></ul><ul><li>Good error messages </li></ul><ul><ul><li>should indicate the problem and explain how to recover </li></ul></ul><ul><li>Help and documentation </li></ul><ul><ul><li>should be concise and easy to search </li></ul></ul>
  12. 12. Norman’s 7 Principles <ul><li>1: Use both knowledge in the world and knowledge in the head. </li></ul><ul><li>2: Simplify the structure of tasks. </li></ul><ul><li>3: Make things visible. </li></ul><ul><li>4: Get the mappings right. </li></ul><ul><li>5: Exploit the power of constraints. </li></ul><ul><li>6: Design for error. </li></ul><ul><li>7: When all else fails, standardise. </li></ul>
  13. 13. Schneiderman’s heuristics (8 Golden Rules) <ul><li>Strive for consistency </li></ul><ul><li>Enable frequent users to use shortcuts </li></ul><ul><li>Offer informative feedback </li></ul><ul><li>Design dialogues to yield closure </li></ul><ul><li>Offer error prevention & simple error handling </li></ul><ul><li>Permit easy reversal of actions </li></ul><ul><li>Support internal locus of control </li></ul><ul><li>Reduce short-term memory load </li></ul><ul><li>(F a ulkner Chapter 7) </li></ul>
  14. 14. Group activity <ul><li>For 3 or 4 of Schneiderman’s heuristics: write a sentence explaining the meaning consider whether the rule holds for XBE, your phone, a game of your choice. </li></ul>
  15. 15. How Many Evaluators? <ul><li>Different people find different problems. </li></ul>
  16. 16. How Many Evaluators?
  17. 17. Think Aloud/Online Self Report “ Think aloud technique is pretty much what it sounds like. You ask someone to do a task, and to think aloud about what they are doing while they are doing it.” - Rugg, 2007
  18. 18. “ The basic concept is simple: you tell the respondent what the task is, and ask them to think aloud while doing it. If they are silent for more than a set length of time (e.g. five seconds) then you use a prearranged prompt to get them talking again” - Rugg, 2007 “ Could you tell me what you’re thinking about now?” “ Are you looking at the background of the picture?” “ Could you tell me what you’re thinking about now?”  “ Are you looking at the background of the picture?” 
  19. 19. “… if you have to transcribe the data, then this can be very time-consuming (in the order of ten hours of transcription per hour of tape, depending on how good your typing is and how loquacious your respondents are).” - Rugg, 2007
  20. 20. “ the convention on transcripts is to use one full stop per second of silence (so “….” shows four seconds of silence). “Um” and “er” sounds are also worth noting, for the same reason, particularly when the respondent is otherwise articulate.” - Rugg, 2007
  21. 21. Qualitative Vs Quantitative <ul><li>Quantitative is data concerning numeric, quantifiable information. </li></ul><ul><li>Qualitative data is verbose, descriptive and more difficult to summarise. </li></ul><ul><li>Techniques such as cluster analysis can help to make meaningful interperetations of qualitative data. </li></ul>