Asking users & experts
The aims <ul><ul><li>Discuss the role of interviews & questionnaires in evaluation. </li></ul></ul><ul><ul><li>Teach basic...
Interviews <ul><li>Unstructured - are not directed by a script. Rich but not replicable.  </li></ul><ul><li>Structured - a...
Basics of interviewing <ul><li>Remember the DECIDE framework </li></ul><ul><li>Goals and questions guide all interviews </...
Things to avoid when preparing interview questions <ul><li>Long questions </li></ul><ul><li>Compound sentences - split int...
Components of an interview <ul><li>Introduction  - introduce yourself, explain the goals of the interview, reassure about ...
The interview process <ul><li>Use the DECIDE framework for guidance </li></ul><ul><li>Dress in a similar way to participan...
Probes and prompts <ul><li>Probes - devices for getting more information. e.g., ‘would you like to add anything?’ </li></u...
Group interviews <ul><li>Also known as ‘focus groups’ </li></ul><ul><li>Typically 3-10 participants </li></ul><ul><li>Prov...
Analyzing interview data <ul><li>Depends on the type of interview </li></ul><ul><li>Structured interviews can be analyzed ...
Questionnaires <ul><li>Questions can be closed or open </li></ul><ul><li>Closed questions are easiest to analyze, and may ...
Questionnaire style <ul><li>Varies according to goal so use the DECIDE framework for guidance </li></ul><ul><li>Questionna...
Developing a questionnaire <ul><li>Provide a clear statement of purpose & guarantee participants anonymity  </li></ul><ul>...
Encouraging a good response <ul><li>Make sure purpose of study is clear </li></ul><ul><li>Promise anonymity </li></ul><ul>...
Advantages of online questionnaires <ul><li>Responses are usually received quickly </li></ul><ul><li>No copying and postag...
Problems with online questionnaires <ul><li>Sampling is problematic if population size is unknown </li></ul><ul><li>Preven...
Questionnaire data analysis & presentation <ul><li>Present results clearly - tables may help </li></ul><ul><li>Simple stat...
Add <ul><li>SUMI </li></ul><ul><li>MUMMS </li></ul><ul><li>QUIS </li></ul>
Asking experts <ul><li>Experts use their knowledge of users & technology to review software usability </li></ul><ul><li>Ex...
Heuristic evaluation <ul><li>Developed Jacob Nielsen in the early 1990s </li></ul><ul><li>Based on heuristics distilled fr...
H O M E R U N <ul><li>H igh-quality content </li></ul><ul><li>O ften updated </li></ul><ul><li>M inimal download time </li...
Nielsen’s heuristics <ul><li>Visibility of system status </li></ul><ul><li>Match between system and real world </li></ul><...
Discount evaluation <ul><li>Heuristic evaluation is referred to as discount evaluation when 5 evaluators are used. </li></...
3 stages for doing heuristic evaluation <ul><li>Briefing session to tell experts what to do </li></ul><ul><li>Evaluation p...
Advantages and problems <ul><li>Few ethical & practical issues to consider </li></ul><ul><li>Can be difficult & expensive ...
Cognitive walkthroughs <ul><li>Focus on ease of learning </li></ul><ul><li>Designer presents an aspect of the design & usa...
The 3 questions <ul><li>Will the correct action be sufficiently evident to the user? </li></ul><ul><li>Will the user notic...
Pluralistic walkthrough <ul><li>Variation on the cognitive walkthrough theme </li></ul><ul><li>Performed by a carefully ma...
Key points <ul><li>Structured, unstructured, semi-structured interviews, focus groups & questionnaires </li></ul><ul><li>C...
Upcoming SlideShare
Loading in …5
×

Chapter13id

589 views

Published on

Published in: Economy & Finance, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
589
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Chapter13id

  1. 1. Asking users & experts
  2. 2. The aims <ul><ul><li>Discuss the role of interviews & questionnaires in evaluation. </li></ul></ul><ul><ul><li>Teach basic questionnaire design. </li></ul></ul><ul><ul><li>Describe how do interviews, heuristic evaluation & walkthroughs. </li></ul></ul><ul><ul><li>Describe how to collect, analyze & present data. </li></ul></ul><ul><ul><li>Discuss strengths & limitations of these techniques </li></ul></ul>
  3. 3. Interviews <ul><li>Unstructured - are not directed by a script. Rich but not replicable. </li></ul><ul><li>Structured - are tightly scripted, often like a questionnaire. Replicable but may lack richness. </li></ul><ul><li>Semi-structured - guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and replicability. </li></ul>
  4. 4. Basics of interviewing <ul><li>Remember the DECIDE framework </li></ul><ul><li>Goals and questions guide all interviews </li></ul><ul><li>Two types of questions: ‘closed questions’ have a predetermined answer format, e.g., ‘yes’ or ‘no’ ‘open questions’ do not have a predetermined format </li></ul><ul><li>Closed questions are quicker and easier to analyze </li></ul>
  5. 5. Things to avoid when preparing interview questions <ul><li>Long questions </li></ul><ul><li>Compound sentences - split into two </li></ul><ul><li>Jargon & language that the interviewee may not understand </li></ul><ul><li>Leading questions that make assumptions e.g., why do you like …? </li></ul><ul><li>Unconscious biases e.g., gender stereotypes </li></ul>
  6. 6. Components of an interview <ul><li>Introduction - introduce yourself, explain the goals of the interview, reassure about the ethical issues, ask to record, present an informed consent form. </li></ul><ul><li>Warm-up - make first questions easy & non-threatening. </li></ul><ul><li>Main body – present questions in a logical order </li></ul><ul><li>A cool-off period - include a few easy questions to defuse tension at the end </li></ul><ul><li>Closure - thank interviewee, signal the end, e.g, switch recorder off. </li></ul>
  7. 7. The interview process <ul><li>Use the DECIDE framework for guidance </li></ul><ul><li>Dress in a similar way to participants </li></ul><ul><li>Check recording equipment in advance </li></ul><ul><li>Devise a system for coding names of participants to preserve confidentiality. </li></ul><ul><li>Be pleasant </li></ul><ul><li>Ask participants to complete an informed consent form </li></ul>
  8. 8. Probes and prompts <ul><li>Probes - devices for getting more information. e.g., ‘would you like to add anything?’ </li></ul><ul><li>Prompts - devices to help interviewee, e.g., help with remembering a name </li></ul><ul><li>Remember that probing and prompting should not create bias. </li></ul><ul><li>Too much can encourage participants to try to guess the answer. </li></ul>
  9. 9. Group interviews <ul><li>Also known as ‘focus groups’ </li></ul><ul><li>Typically 3-10 participants </li></ul><ul><li>Provide a diverse range of opinions </li></ul><ul><li>Need to be managed to: - ensure everyone contributes - discussion isn’t dominated by one person - the agenda of topics is covered </li></ul>
  10. 10. Analyzing interview data <ul><li>Depends on the type of interview </li></ul><ul><li>Structured interviews can be analyzed like questionnaires </li></ul><ul><li>Unstructured interviews generate data like that from participant observation </li></ul><ul><li>It is best to analyze unstructured interviews as soon as possible to identify topics and themes from the data </li></ul><ul><li>(refer to page 395) </li></ul>
  11. 11. Questionnaires <ul><li>Questions can be closed or open </li></ul><ul><li>Closed questions are easiest to analyze, and may be done by computer </li></ul><ul><li>Can be administered to large populations </li></ul><ul><li>Paper, email & the web used for dissemination </li></ul><ul><li>Advantage of electronic questionnaires is that data goes into a data base & is easy to analyze </li></ul><ul><li>Sampling can be a problem when the size of a population is unknown as is common online </li></ul>
  12. 12. Questionnaire style <ul><li>Varies according to goal so use the DECIDE framework for guidance </li></ul><ul><li>Questionnaire format can include: - ‘yes’, ‘no’ checkboxes - checkboxes that offer many options - Likert rating scales - semantic scales - open-ended responses </li></ul><ul><li>Likert scales have a range of points </li></ul><ul><li>3, 5, 7 & 9 point scales are common </li></ul><ul><li>Debate about which is best </li></ul>
  13. 13. Developing a questionnaire <ul><li>Provide a clear statement of purpose & guarantee participants anonymity </li></ul><ul><li>Plan questions - if developing a web-based questionnaire, design off-line first </li></ul><ul><li>Decide on whether phrases will all be positive, all negative or mixed </li></ul><ul><li>Pilot test questions - are they clear, is there sufficient space for responses </li></ul><ul><li>Decide how data will be analyzed & consult a statistician if necessary </li></ul>
  14. 14. Encouraging a good response <ul><li>Make sure purpose of study is clear </li></ul><ul><li>Promise anonymity </li></ul><ul><li>Ensure questionnaire is well designed </li></ul><ul><li>Offer a short version for those who do not have time to complete a long questionnaire </li></ul><ul><li>If mailed, include a s.a.e. </li></ul><ul><li>Follow-up with emails, phone calls, letters </li></ul><ul><li>Provide an incentive </li></ul><ul><li>40% response rate is high, 20% is often acceptable </li></ul>
  15. 15. Advantages of online questionnaires <ul><li>Responses are usually received quickly </li></ul><ul><li>No copying and postage costs </li></ul><ul><li>Data can be collected in database for analysis </li></ul><ul><li>Time required for data analysis is reduced </li></ul><ul><li>Errors can be corrected easily </li></ul><ul><li>Disadvantage - sampling problematic if population size unknown </li></ul><ul><li>Disadvantage - preventing individuals from responding more than once </li></ul>
  16. 16. Problems with online questionnaires <ul><li>Sampling is problematic if population size is unknown </li></ul><ul><li>Preventing individuals from responding more than once </li></ul><ul><li>Individuals have also been known to change questions in email questionnaires </li></ul>
  17. 17. Questionnaire data analysis & presentation <ul><li>Present results clearly - tables may help </li></ul><ul><li>Simple statistics can say a lot, e.g., mean, median, mode, standard deviation </li></ul><ul><li>Percentages are useful but give population size </li></ul><ul><li>Bar graphs show categorical data well </li></ul><ul><li>More advanced statistics can be used if needed </li></ul>
  18. 18. Add <ul><li>SUMI </li></ul><ul><li>MUMMS </li></ul><ul><li>QUIS </li></ul>
  19. 19. Asking experts <ul><li>Experts use their knowledge of users & technology to review software usability </li></ul><ul><li>Expert critiques (crits) can be formal or informal reports </li></ul><ul><li>Heuristic evaluation is a review guided by a set of heuristics </li></ul><ul><li>Walkthroughs involve stepping through a pre-planned scenario noting potential problems </li></ul>
  20. 20. Heuristic evaluation <ul><li>Developed Jacob Nielsen in the early 1990s </li></ul><ul><li>Based on heuristics distilled from an empirical analysis of 249 usability problems </li></ul><ul><li>These heuristics have been revised for current technology, e.g., HOMERUN for evaluating commercial websites </li></ul><ul><li>Heuristics still needed for mobile devices, wearables, virtual worlds, etc. </li></ul><ul><li>Design guidelines form a basis for developing heuristics </li></ul>
  21. 21. H O M E R U N <ul><li>H igh-quality content </li></ul><ul><li>O ften updated </li></ul><ul><li>M inimal download time </li></ul><ul><li>E ase of use </li></ul><ul><li>R elevant to users’ needs </li></ul><ul><li>U nique to the online medium </li></ul><ul><li>N etcentric corporate culture </li></ul><ul><li>Nielsen (1999) </li></ul>
  22. 22. Nielsen’s heuristics <ul><li>Visibility of system status </li></ul><ul><li>Match between system and real world </li></ul><ul><li>User control and freedom </li></ul><ul><li>Consistency and standards </li></ul><ul><li>Help users recognize, diagnose, recover from errors </li></ul><ul><li>Error prevention </li></ul><ul><li>Recognition rather than recall </li></ul><ul><li>Flexibility and efficiency of use </li></ul><ul><li>Aesthetic and minimalist design </li></ul><ul><li>Help and documentation </li></ul>
  23. 23. Discount evaluation <ul><li>Heuristic evaluation is referred to as discount evaluation when 5 evaluators are used. </li></ul><ul><li>Empirical evidence suggests that on average 5 evaluators identify 75-80% of usability problems. </li></ul>
  24. 24. 3 stages for doing heuristic evaluation <ul><li>Briefing session to tell experts what to do </li></ul><ul><li>Evaluation period of 1-2 hours in which: - Each expert works separately - Take one pass to get a feel for the product - Take a second pass to focus on specific features </li></ul><ul><li>Debriefing session in which experts work together to prioritize problems </li></ul>
  25. 25. Advantages and problems <ul><li>Few ethical & practical issues to consider </li></ul><ul><li>Can be difficult & expensive to find experts </li></ul><ul><li>Best experts have knowledge of application domain & users </li></ul><ul><li>Biggest problems - important problems may get missed - many trivial problems are often identified </li></ul>
  26. 26. Cognitive walkthroughs <ul><li>Focus on ease of learning </li></ul><ul><li>Designer presents an aspect of the design & usage scenarios </li></ul><ul><li>One of more experts walk through the design prototype with the scenario </li></ul><ul><li>Expert is told the assumptions about user population, context of use, task details </li></ul><ul><li>Experts are guided by 3 questions </li></ul>
  27. 27. The 3 questions <ul><li>Will the correct action be sufficiently evident to the user? </li></ul><ul><li>Will the user notice that the correct action is available? </li></ul><ul><li>Will the user associate and interpret the response from the action correctly? As the experts work through the scenario they note problems </li></ul>
  28. 28. Pluralistic walkthrough <ul><li>Variation on the cognitive walkthrough theme </li></ul><ul><li>Performed by a carefully managed team </li></ul><ul><li>The panel of experts begins by working separately </li></ul><ul><li>Then there is managed discussion that leads to agreed decisions </li></ul><ul><li>The approach lends itself well to participatory design </li></ul>
  29. 29. Key points <ul><li>Structured, unstructured, semi-structured interviews, focus groups & questionnaires </li></ul><ul><li>Closed questions are easiest to analyze & can be replicated </li></ul><ul><li>Open questions are richer </li></ul><ul><li>Check boxes, Likert & semantic scales </li></ul><ul><li>Expert evaluation: heuristic & walkthroughs </li></ul><ul><li>Relatively inexpensive because no users </li></ul><ul><li>Heuristic evaluation relatively easy to learn </li></ul><ul><li>May miss key problems & identify false ones </li></ul>

×