My Works About UT Plan (draft) 2006


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

My Works About UT Plan (draft) 2006

  1. 1. Usability Evaluation Plan For Kijiji Taiwan Prepared for: Sydney Chen / Head of Kijiji TW Tang Hsun / Director of User Experience Design Prepared by: wwwins consulting Author: Ruby Kuo & Alex Chiu Date: Sep. 20, 2006 Revision 2.0 Approvals: ______________________ _______________________ Name Name
  2. 2. Usability Evaluation Plan for Kijiji TW 2006/9/20 0.Introduction This document describes the usability evaluation plan for Kijiji TW. The purpose of this study is to discover significant usability issues by testing real user with current live web site. This plan includes the following sections: 1. Purpose of the usability evaluation 2. Participants 3. Design of the usability evaluation 4. Data collection methodology 5. Deliverables 6. Schedule 7. Appendix - Script of test scenario (English version) Background information of Kijiji TW Kijiji Taiwan was launched in May, 2005. The site is a group of free, local, community classifieds websites. It offers a convenient, fun, and easy way (no need to register as a member to post or respond an ad) for people in the same city to meet, trade, share ideas, and help each other out in areas such as goods, cars, services, housing, jobs, and personals. There are over a hundred thousand ads now, and the average increase rate is about 30%. Some most popular categories (count by the number of ad post) are like: [1]housing (租屋售屋), [2]service(個人服務/商家), [3]flea market (跳蚤市場), and [4]jobs(當地求職求才). Moreover, Kijiji cooperates with urMap to supply the map exploration and quest function to assist user navigate efficiently. As realized the Kijiji TW target user profile can be described as below: busy working adults age 20-40 Internet daily usage 2-8 hrs per day Trust individuals & turn to internet community for help Use internet to connect to their lives and friends Proactive, independent SOHOs or small shops owners Page 1
  3. 3. Usability Evaluation Plan for Kijiji TW 2006/9/20 1. Purpose of the Usability Evaluation 1.1. Concerns This section outlines the specific concerns that refine from the objectives in RFP document from Kijiji TW. The particular questions to be addressed in this study include: Overall O1). Will users be able to use this website easily, efficiently and satisfyingly? O2). Is the navigation concept clear? Will users always know where they are and easily find their way around the website? O3). Is the information architecture about classified ad appropriate? Will users be able to find the information or function they need very easy and quickly? O4). Will users be able to understand the category naming or labeling meaning? Are they meeting users’ expectation? O5). Are the website’s color, graphics and design pleasing and attractive? O6). What’s the user’s preference of this website? What is the overall impression of it, and what emotional responses towards it? O7). Which contents (information, functions, and services) meet users’ needs? And what are their wishes in the future? O8). Will users regard this website useful for them? Will they return in the future on their own? Or recommend it to other friends? If not, what are the reasons? Figure it out. Specific task and function S1). Do users understand and use the "Location" options accurately during this study? S2). Are users able to leave “Comment” on an Ad, and do they regard it useful or confusing (compared with replying via email to the Poster of the Ad)? S3). How do users search for “Housing (租屋售屋)” on the site? Do they need additional tools or information? S4). How do users search for “Services(個人服務/商家)” on the site? Do they need additional tools or information? S5). Do users understand the concept of Wanted/Offer(尋找、徵求/出售、提供) on the site? Is this clear? Is it useful? S6). Do users use the Maps available on the site? If so, when will they use it, for what purpose and how? S7). Do users have any difficulties in selecting a category to post (or browse) an Ad? If so, what are they? Additional A1). Will the concerns listed above get any different opinions (or findings) between existing and new kijiji users? If so, what’s the difference? A2). To understand how people commuting between two cities feel when they have to select different subsites for viewing different regions’ ads. A3). How do users feel about my Kijiji? Page 2
  4. 4. Usability Evaluation Plan for Kijiji TW 2006/9/20 3.1.3. Performance and subjective preference evaluation The performance evaluation consists of a series of tasks that are evaluated separately and sequentially. The typically procedure is as follows: • After the orientation, the participants will be asked to sit down at the computer. The facilitator will give the participants the task scenario script. • Every participant will be encouraged to work without guidance, and be asked “thinking aloud” while performing the tasks of the test. • The facilitator may ask the participant to verbalize his/her thoughts if the participant becomes stuck, confused or in silence. • The after-scenario questionnaires for collecting subjective preference data (when available). 3.1.4. Post-test stage At this stage, each participant will be interviewed by the facilitator, and assignment will include the following: • Completion of a post-questionnaire in which the participants share their opinions on the website’s usability, appearance and general impressions of it (such as, look and feel and navigation design). • Participant’s overall comments about his or her experience. • Clarify and collect more insight or internal ideas of his/her manipulate behavior. 4. Data Collection Methodology Data will be collected through the use of a “thinking aloud” protocol and videotaping. Observer’s notes will also assist for data analysis (include oral data, facial expression, emotional expression, manipulate behavior and body language). Measures to be collected include the following: Performance measure: 1. Time/percentage to complete each scenario session & task. 2. The number of non-complete scenario sessions & tasks. 3. The number of all correct/incorrect selections (in each scenario & task). i、 The number of incorrect menu or icons choices (in each scenario & task). ii、 The number of total clicks (in each scenario & task). 4. The number of times to use “search” function (and first/each use in which scenario). 5. The number of times to use “map” function (and first/each use in which scenario). 6. The number of positive/critical statements (in each scenario). 7. The number of indications of frustration/joy from the participant (in each scenario). Subjective preference measure: Page 7
  5. 5. Usability Evaluation Plan for Kijiji TW 2006/9/20 1. Degree of “ease of use” overall (and/or in each scenario). 2. Degree of “ease of learning” overall (and/or in each scenario). 3. Degree of “satisfaction” overall (and/or in each scenario). 4. Degree of “usefulness” overall (and/or in each scenario). 5. Confidence with their performance overall (and/or in each scenario). 6. Degree of awareness where they are and ability of way finding. 7. Preference of visual design overall (include color, graphics, look and feel) 8. The most attractive function, service or content. 9. The number of subjective opinions of the usability and aesthetics of the website expressed by the participants. 10. Degree of “intention” that returns and recommend to friends. 5. Deliverables 1. PowerPoint presentation with summary, interpretations, recommendations, and embedded video clips to illustrate key findings. 2. Video/audio recordings of each test sessions. 3. Final presentation via telephone conference. 4. Digital audio files of simultaneous translation of each test sessions. 6. Schedule The usability evaluation will be conducted between 25 Sep. and 1 Oct. 2006. The final detail schedule is dependent on approval of this evaluation plan by Kijiji TW and the selection of qualified participants. Page 8
  6. 6. Usability Evaluation Plan for Kijiji TW 2006/9/20 7. Appendix A. Script of test scenario (English version) Participant greeting and background questionnaire (5mins) Hi, welcome, thanks for coming….. I am ____ (facilitator introduce itself ) To break the ice - based on the basic information we have to chat with participant, and to make them comfortable… Pre test-questionnaire – Give them the agreement and background questionnaire to fill out… Orientation (5mins) Give an introduction of this study – the purpose, process and evaluation time for this test. Today we are asking you to serve as an evaluator of this Web site and to complete a set of scenarios. Our goal is to see how easy or difficult you find the site to use. We will record your reactions and opinions; so, we may ask you to clarify statements that you make from time to time. There are totally 8 scenarios that we are going to take later, we calculate that will need around an hour to finish. And I’m here to assist and give you orientation in each task scenario. For example, I will ask you perform a specific task in a website, and would like you to think a loud while you look for the answer. Although you may ask question anytime, but during the task session I will not be able to offer any suggestions or hints. Don’t worry if you can’t find the answer every time. Remember, we are testing the effectiveness of website--this is not a test of you! When you are ready, than we can begin. And when you feel that you need to take a break during this study, please say so. Here are couples thing that I like you keep in mind: This is not a test of you; you’re testing the site. So don’t worry about making mistakes. There is no right or wrong answer. We really just want to know if we designed the site well for you. If you ever feel that you are lost or cannot complete a scenario with the information that you have been given, please let me know. I’ll ask you what you might do in a real-world setting and then either put you on the right track or move you on to the next scenario. We will be video recording this session for further study if needed. Your name will not be associated or reported with data or findings from this evaluation. Please fill out the video release form. Finally, as you use the site, please do so as you would at home or your office. I do ask that when looking for information, you do so as quickly and as accurately as you can. Performance and subjective preference evaluation (60~80 mins) Scenario 1. Warm-up session. To learn the prior user experience of group A and the first impression of Kijiji Taiwan homepage from group B. (to group A) (to group B) 1.1 Please show me how you use kijiji 1.1 I will show you a homepage of a website usually. (Open kijiji Taiwan hompage) we are testing today. Please look at it for 5-10 seconds as you might did when Items to note in task 1.1: visiting a website for the first time. Then a) What tasks they perform most often? And how? describe what you have seen. Items to note in task 1.1: Page 9