• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Analysis in usability evaluation: Questionnaire
 

Analysis in usability evaluation: Questionnaire

on

  • 970 views

Questionnaire used in questionnaire survey study of analysis in usability evaluation practice. Data collected spring 2011.

Questionnaire used in questionnaire survey study of analysis in usability evaluation practice. Data collected spring 2011.

Statistics

Views

Total Views
970
Views on SlideShare
970
Embed Views
0

Actions

Likes
0
Downloads
6
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Analysis in usability evaluation: Questionnaire Analysis in usability evaluation: Questionnaire Document Transcript

    • Analysis in usability evaluationQuestionnaire to respondents reportingon USABILITY TESTING
    •  Intro Welcome to our study on how usability professionals conduct the later phase of usability evaluation.  For usability testing, the later phase is the process of going from user observations to the results hand­over to the  client. For usability inspection, the process from walking through the user interface to the results hand­over.  We invite usability practitioners who have conducted a usability evaluation within the last six months.  This study is important. Currently we do not have a sufficient understanding of how the later phases of usability  evaluation is conducted. The results will provide valuable guidance for future research on usability evaluation  methods.  Your participation will take about 15 minutes. As an appreciation of your effort, we draw a prize between the survey participants – the winner can choose between an iPod Touch 32 GB or a 250$ (US) Amazon gift card.. You will also be given free access to the study report prior to its publication.  The questionnaire consists of 32 questions, mainly targeting your latest usability evaluation.  Other  It is important for the quality of the study that you answer honestly. All responses are collected anonymously and  cannot be tracked back to you. To participate in the drawing of the prize you have to leave your e­mail address, but  the address will be fully disconnected from your other responses. You are free to resign from the study at any time.  Thanks for participating!  (Study organiser names removed)   
    •  Your latest usability evaluation*1. Yes, I have conducted a usability evaluation within the last six months and would like to partiMy latest usability evaluation was:   j k l m n Usability test (evaluation that involves observation of users) Other    j k l m n Usability inspection (analytical evaluation)   j k l m n Have not conducted a usability evaluation within the last six months  
    •  About you2. How many years have you been working as a usability practitioner?  3. How many usability evaluations have you conducted the last 12 months?(approximately)  4. What kind of usability projects are you usually engaged in?Other    j k l m n Inhouse development projects   j k l m n Consultancy / development projects for external clients   j k l m n Adaptation of commercial software   j k l m n Scientific research projects   j k l m n Other (please specify)  5. Your educational background (subject/field, level)  6. Your country of work.    
    •  Your latest usability test The remainder of the questionnaire will target your latest usability test. Please make sure that you have one particular test in mind when answering the remaining questions. 7. When did you start working on your latest usability test?   j k l m n More than 6 months ago   j k l m n 3­6 months ago   j k l m n 1­2 months ago   j k l m n Less than 1 month ago8. In which development phase was the object of evaluation?   j k l m n Concept (overall presentation of solution but without presentation of interaction mechanisms)   j k l m n Early prototype   j k l m n Advanced prototype   j k l m n System in use9. How many user participants were involved? Other   10. How many hours were spent on preparing, conducting and reporting the usabilitytest(Approximately, in total for all usability professionals involved. Just leave blank if youdo not know.)  11. Would you say this usability test was typical with respect to time and resourceconstraints?   j k l m n Yes   j k l m n No, we had less time and/or resources than we typically have   j k l m n No, we had more time and/or resources than we typically have  
    •  Perceptions of your latest usability test12. How challenging did you consider the following when conducting this usability test? Not at all  Little  Somewhat  Other  Challenging Very challenging challenging challenging challenging To identify all relevant usability problems j k l m n j k l m n j k l m n j k l m n j k l m n To avoid "false" usability problems (that is, issues that seem to be  j k l m n j k l m n j k l m n j k l m n j k l m n a problem during evaluation but that turns out not to be a problem  in the real world context). To make good redesign suggestions j k l m n j k l m n j k l m n j k l m n j k l m n To prioritize the problems and/or redesign suggestions j k l m n j k l m n j k l m n j k l m n j k l m n To convince the development team and/or customer j k l m n j k l m n j k l m n j k l m n j k l m n13. Please indicate your agreement with the following statements Strongly  Disagree Neither Agree Strongly agree disagree My latest usability test was highly successful j k l m n j k l m n j k l m n j k l m n j k l m n My latest usability inspection has caused important changes to the  j k l m n j k l m n j k l m n j k l m n j k l m n solution, or will so in the immediate future My latest usability test has lead to extensive dialogue on how to  j k l m n j k l m n j k l m n j k l m n j k l m n improve the design The results of my latest usability test have been actively used in  j k l m n j k l m n j k l m n j k l m n j k l m n the development project It was easy to communicate the results of my latest usability test to  j k l m n j k l m n j k l m n j k l m n j k l m n the people in the development project  
    •  During the user sessions14. What kind of notes was taken during the user sessions of your latest usability test?(Choose one or more)   c d e f g Informal notes   c d e f g Notes on simple forms (for example forms structured as a few high­level sections)   c d e f g Notes on detailed forms (for example forms including tables or similar for filling in user data) If you used simple or detailed forms for note taking, please describe the form briefly.  5 615. Were the user participants or the observers asked for their opinion on …(choose one or more for each column) Other  User participants Observers Possible usability problems c d e f g c d e f g Possible redesign suggestions c d e f g c d e f g Other issues (please specify) c d e f g c d e f g None of the above c d e f g c d e f gIf you answered "Other", please specify here   
    •  After the user sessions ­ I16. Did you do a summary of key findings immediately following each test person?   j k l m n Yes, the present observers and usability personnel summarized key findings together immediately following each test session   j k l m n Yes, I summarized key findings by myself immediately following each test person   j k l m n No, but the present observers and usability personnel summarized key findings at the end of the day   j k l m n No, but I summarized key findings by myself at the end of the day   j k l m n No such immediate summary of key findings was conducted.17. Did you review video recordings from the user sessions of your latest usability test?   j k l m n No recordings were made   j k l m n Recordings were made, but not reviewed   j k l m n Recordings were made, but reviewed in part only   j k l m n Recordings were made and fully reviewed18. Which of the following resources did you make explicit use of during analysis in your latest ustest?(Choose one or more)   c d e f g My professional experience   c d e f g Previously established personas   c d e f g Heuristics or guidelines   c d e f g Design patterns   c d e f g Standards (norm or technical requirement established within a given field or application area)   c d e f g Other resources  19. Which measures, if any, did you use during analysis in your latest usability test?   c d e f g Task completion   c d e f g Task time   c d e f g Error rate   c d e f g Satisfaction   c d e f g Other (please specify)    
    •  After the user sessions ­ II20. Which tools, if any, did you use for usability problem description and analysis inyour latest usability test? 5 6  21. Please provide a brief description of how you decided on something being ausability problem. 5 6  22. If an incident was observed with only one of the users participating in your latestusability test,how did you decide whether this was a usability problem or not? 5 6    
    •  Collaboration in your latest usability test Other 23. What kind of collaboration did you have with other usability professionals duringthe analysisof your latest usability test? (Choose one or more)   c d e f g Short discussions immediately following user test sessions   c d e f g Review of draft report   c d e f g Guidance when conducting analysis   c d e f g Individual analysis of data from different user sessions by myself and at least one other colleague   g Individual analysis of data from the same user sessions by myself and at least one other colleague c d e f   c d e f g The data analysis was conducted as a group activity by two or more usability professionals.   g No collaboration c d e f  
    •  Motivation for collaboration24. If you were to choose one, what would you say was the main purpose of collaboration inthe period between user test sessions and the final report? (Choose one)   j k l m n Identify more problems   j k l m n Generate better redesign suggestions   j k l m n Quality assurance   j k l m n Improve reliability / avoid that the findings were biased by personal perspective   j k l m n Other (please specify)    
    •  The usability test deliverable25. How would you characterize the deliverable from your latest usability test?   j k l m n A set of redesign suggestions in part motivated from usability problems   j k l m n A set of redesign suggestions in response to a set of usability problems   j k l m n A set of usability problems with some redesign suggestions   j k l m n A set of usability problems with no redesign suggestions  
    •  Redesign suggestions26. When did you make the redesign suggestions?   j k l m n First all usability problems were identified, then the redesign suggestions were made Other    j k l m n Whenever a usability problem was identified, a redesign suggestion was immediately made j k l m n Some redesign suggestions were immediately when a usability problem was identified, others were made only when all usability  problems had been identified    j k l m n Other (please specify) 5 6  27. How did you reach the redesign suggestions that were included in the deliverable?(Chose one or more)   c d e f g In response to usability problems   c d e f g On basis of recognition that the chosen solution was not optimal, even though no usability problem had been observed.   c d e f g Test participant suggestions   c d e f g Observer suggestions   c d e f g Client suggestions   c d e f g Other (please specify) 5 6  28. How did you present redesign suggestions in the deliverable? (Chose one or more)   c d e f g Textual descriptions   c d e f g Annotated screen shots   c d e f g Sketching   c d e f g UI digital mock­up   Other  c d e f g Graphical elements or code that could be immediately implemented   c d e f g Oral presentation   c d e f g Other (please specify) 5 6    
    •  Structured formats and severity classifications29. Did you use screen­shots to illustrate usability problem descriptions?   j k l m n No Other    j k l m n Yes, for some of the problems   j k l m n Yes, for most or all of the problems30. Did you use a structured format for usability problem description?   j k l m n No, the problems were described in plain prose.   j k l m n Yes, the problems were described according to my/our own format   j k l m n Yes, the problems were described according to a structured format described in standards or literature If you used a structured format described in standards or literature, please specify 31. Did you prioritize the usability problems or redesign suggestions?   j k l m n Yes, but only usability problems Other    j k l m n Yes, but only redesign suggestions   j k l m n Yes, both usability problems and redesign suggestions   j k l m n No, neither usability problems nor redesign suggestions were prioritized  
    •  Type of ratings32. How did you prioritize the usability problems or redesign suggestions in your latestusability test? (Choose the alternative closest to describing the classification you used)   j k l m n Severity ratings (classifying the predicted impact of a usability problem on user behavior)   j k l m n Urgency ratings (classifying how important it is to make a design change in response to a usability problem or redesign suggestion)   j k l m n Other (please specify)    
    •  Thank youThank you for participating! This is a valuable contribution to the research on analysis in usability evaluation. For distribution of report results, and for notification if you win the participant prize, please provide your e­mail address on the next page. The answers you have given on the previous pages are anonymous even if you leave your e­mail address. The e­mail address will at no point in time be connected to your answers. 
    • Analysis in usability evaluationQuestionnaire to respondents reportingon USABILITY INSPECTION
    •  Intro Welcome to our study on how usability professionals conduct the later phase of usability evaluation.  Other  For usability testing, the later phase is the process of going from user observations to the results hand­over to the  client. For usability inspection, the process from walking through the user interface to the results hand­over.  We invite usability practitioners who have conducted a usability evaluation within the last six months.  This study is important. Currently we do not have a sufficient understanding of how the later phases of usability  evaluation is conducted. The results will provide valuable guidance for future research on usability evaluation  methods.  Your participation will take about 15 minutes. As an appreciation of your effort, we draw a prize between the survey participants – the winner can choose between an iPod Touch 32 GB or a 250$ (US) Amazon gift card.. You will also be given free access to the study report prior to its publication.  The questionnaire consists of 32 questions, mainly targeting your latest usability evaluation.  It is important for the quality of the study that you answer honestly. All responses are collected anonymously and  Other  cannot be tracked back to you. To participate in the drawing of the prize you have to leave your e­mail address, but  the address will be fully disconnected from your other responses. You are free to resign from the study at any time.  Thanks for participating!  (Study organizer names removed)   
    •  Your latest usability evaluation*1. Yes, I have conducted a usability evaluation within the last six months and would like to partiMy latest usability evaluation was:   j k l m n Usability test (evaluation that involves observation of users)   j k l m n Usability inspection (analytical evaluation)   j k l m n Have not conducted a usability evaluation within the last six months  
    •  ABOUT YOU2. How many years have you been working as a usability practitioner?  3. How many usability evaluations have you conducted the last 12 months?(approximately)  4. What kind of usability projects are you usually engaged in?   j k l m n Inhouse development projects   j k l m n Consultancy / development projects for external clients   j k l m n Adaptation of commercial software   j k l m n Scientific research projects   j k l m n Other (please specify)  5. Your educational background (subject/field, level)  Other 6. Your country of work.    
    •  YOUR LATEST USABILITY INSPECION The remainder of the questionnaire will target your latest usability inspection (analytical evaluation). Please make sure that you have one particular inspection in mind when answering the remaining questions. 7. When did you start working on your latest usability inspection?   j k l m n More than 6 months ago   j k l m n 3­6 months ago   j k l m n 1­2 months ago   j k l m n Less than 1 month ago8. In which development phase was the object of evaluation?   j k l m n Concept (overall presentation of solution but without presentation of interaction mechanisms)   j k l m n Early prototype   j k l m n Advanced prototype   j k l m n System in use9. Which usability inspection method was used?   j k l m n Heuristic evaluation   j k l m n Cognitive walkthrough   j k l m n Informal expert evaluation   j k l m n No particular method / mix of methods   j k l m n Other method (please specify)  10. Please provide a brief description of the inspection method. 5 6  11. How many hours were spent on preparing, conducting and reporting the usabilityinspection(Approximately, in total for all usability professionals involved. Just leave blank if youdo not know.)  12. Would you say this usability inspection was typical with respect to time andresource constraints?   j k l m n Yes   j k l m n No, we had less time and/or resources than we typically have   j k l m n No, we had more time and/or resources than we typically have  
    •  PERCEPTIONS OF YOUR LATEST USABILITY INSPECTION13. How challenging did you consider the following when conducting this usability inspection? Not at all  Little  Somewhat  Challenging Very challenging challenging challenging challenging To identify all relevant usability problems j k l m n j k l m n j k l m n j k l m n j k l m n To avoid "false" usability problems (that is, issues that seem to be  j k l m n j k l m n j k l m n j k l m n j k l m n a problem during evaluation but that turns out not to be a problem  in the real world context). To make good redesign suggestions j k l m n j k l m n j k l m n j k l m n j k l m n To prioritize the problems and/or redesign suggestions j k l m n j k l m n j k l m n j k l m n j k l m n To convince the development team and/or customer j k l m n j k l m n j k l m n j k l m n j k l m n14. Please indicate your agreement with the following statements Strongly  Disagree Neither Agree Strongly agree disagree My latest usability inspection was highly successful j k l m n j k l m n j k l m n j k l m n j k l m n My latest usability inspection has caused important changes to the  j k l m n j k l m n j k l m n j k l m n j k l m n solution, or will so in the immediate future My latest usability inspection has lead to extensive dialogue on  j k l m n j k l m n j k l m n j k l m n j k l m n how to improve the design The results of my latest usability inspection have been actively  j k l m n j k l m n j k l m n j k l m n j k l m n used in the development project It was easy to communicate the results of my latest usability  j k l m n j k l m n j k l m n j k l m n j k l m n inspection to the people in the development project  
    •  DURING THE INSPECTION (1)15. What kind of notes was taken when going through the object of evaluation duringyour latest usability inspection?(Choose one or more)   c d e f g Informal notes   c d e f g Notes on simple forms (for example forms structured as a few high­level sections)   c d e f g Notes on detailed forms (for example forms including tables or similar for filling in user data) If you used simple or detailed forms for note taking, please describe the form briefly.  5 616. Which of the following resources did you make explicit use of during analysis in your latest usinspection?(Choose one or more)   c d e f g My professional experience   c d e f g Previously established personas   c d e f g Heuristics or guidelines   c d e f g Design patterns   c d e f g Standards (norm or technical requirement established within a given field or application area)   c d e f g Previously established scenarios of use   c d e f g Other resources    
    •  DURING THE INSPECTION (2)17. Which tools, if any, did you use for usability problem description and analysis inyour latest usability inspection? 5 6  18. Please provide a brief description of how you decided on something being ausability problem. 5 6    
    •  COLLABORATION IN YOUR LATEST USABILITY INSPECTION19. What kind of collaboration did you have with other usability professionals duringyour latest usability inspection? (Choose one or more)   c d e f g Short discussions immediately following startup of the inspection   c d e f g Review of draft report   c d e f g Guidance when conducting the inspection   c d e f g Individual inspections of different parts of the system by myself and at least one other colleague   g Individual inspections of the same part of the system by myself and at least one other colleague c d e f   c d e f g The inspection was conducted as a group activity by two or more usability professionals.   g No collaboration c d e f  
    •  MOTIVATION FOR COLLABORATION20. If you were to choose one, what would you say was the main purpose of collaboration inthe later phases of your latest usability inspection? (Choose one)   j k l m n Identify more problems   j k l m n Generate better redesign suggestions   j k l m n Quality assurance   j k l m n Improve reliability / avoid that the findings were biased by personal perspective   j k l m n Other (please specify)  21. If only one of the usability personnel participating in your latest usability inspectionidentified something as a usability problem, how did you decide whether this was to bereported as a usability problem or not? 5 6    
    •  THE USABILITY INSPECTION DELIVERABLE22. How would you characterize the deliverable from your latest usability inspection?   j k l m n A set of redesign suggestions in part motivated from usability problems   j k l m n A set of redesign suggestions in response to a set of usability problems   j k l m n A set of usability problems with some redesign suggestions   j k l m n A set of usability problems with no redesign suggestions  
    •  REDESIGN SUGGESTIONS23. When did you make the redesign suggestions?   j k l m n First all usability problems were identified, then the redesign suggestions were made   j k l m n Whenever a usability problem was identified, a redesign suggestion was immediately made j k l m n Some redesign suggestions were immediately when a usability problem was identified, others were made only when all usability problems had been identified    j k l m n Other (please specify) 5 6  24. How did you reach the redesign suggestions that were included in the deliverable?(Chose one or more)   c d e f g In response to usability problems   c d e f g On basis of recognition that the chosen solution was not optimal, even though no usability problem had been observed.   c d e f g Client suggestions   c d e f g Colleague or peer suggestions   c d e f g Other (please specify) 5 6  25. How did you present redesign suggestions in the deliverable? (Chose one or more)   c d e f g Textual descriptions   c d e f g Annotated screen shots   c d e f g Sketching   c d e f g UI digital mock­up   c d e f g Graphical elements or code that could be immediately implemented   c d e f g Oral presentation   c d e f g Other (please specify) 5 6    
    •  STRUCTURED FORMATS AND SEVERITY CLASSIFICATIONS26. Did you use screen­shots to illustrate usability problem descriptions?   j k l m n No   j k l m n Yes, for some of the problems   j k l m n Yes, for most or all of the problems27. Did you use a structured format for usability problem description?   j k l m n No, the problems were described in plain prose.   j k l m n Yes, the problems were described according to my/our own format   j k l m n Yes, the problems were described according to a structured format described in standards or literatureIf you used a structured format described in standards or literature, please specify 28. Did you prioritize the usability problems or redesign suggestions?   j k l m n Yes, but only usability problems   j k l m n Yes, but only redesign suggestions   j k l m n Yes, both usability problems and redesign suggestions   j k l m n No, neither usability problems nor redesign suggestions were prioritized  
    •  TYPE OF RATINGS29. How did you prioritize the usability problems or redesign suggestions in your latestusability inspection? (Choose the alternative closest to describing the classificationyou used)   j k l m n Severity ratings (classifying the predicted impact of a usability problem on user behavior)   j k l m n Urgency ratings (classifying how important it is to make a design change in response to a usability problem or redesign suggestion)   j k l m n Other (please specify)    
    •  Thank youThank you for participating! This is a valuable contribution to the research on analysis in usability evaluation. For distribution of report results, and for notification if you win the participant prize, please provide your e­mail address on the next page. The answers you have given on the previous pages are anonymous even if you leave your e­mail address. The e­mail address will at no point in time be connected to your answers.