Cheap& Quick Internal user testing

Cheap & Quick Internal User Testing - Flirtomatic’s Recent Challenge - 3 Dec 2008 Jaeseung Chang (jaeseung@flirtomatic.com) Usability Architect @ Handmade Mobile Entertainment Ltd. 1 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

Quick Introduction - BA in Education from SNU - MSc in Information Technology from QMUL - PgDip in HCI from UCLIC - MSc by Research in Computer Science from QMUL - Open Education (Korea) / Usability Designer - Samsung Electronics (Headquarter in Korea) / Mobile Usability Designer + Researcher - Handmade Mobile (Flirtomatic) / Mobile Usability Architect 2 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

Cheap & Quick Internal User Testing – Flirtomatic’s Recent Challenge 1. Why Cheap & Quick? 2. Experiment Design Good Lessons – Dos and Donts 3. 3 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

1. Why Cheap & Quick? 1.1. Evaluation Needs - We all know the necessity of usability evaluation - At least 2 times per 1 iteration cycle - But the reality sometimes doesn’t help Requirement Fast Analysis Prototyping Evaluation Evaluation Specification Implementation 4 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

1. Why Cheap & Quick? 1.2. Usability Firms - Hiring professional research firms? - Excellent facilities, professional experiment design, reliable recruitment, & nice report (maybe?) - But costs a fortune! 5 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

1. Why Cheap & Quick? 1.3. Self-Evaluation - Then we should go DIY - Venue: How about using our meeting room? - Experiment design: We can do that maybe? - Recruitment: Find some volunteers from our users? - Observation and analysis?: 15 day trial version MORAE! 6 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

2. Experiment Design 2.1. Overview - The 1st iteration of our ‘Redesign Project’ - Based on wireframes (workflows) produced so far - Evaluation on our first redesign UI/IA works - To discover good/bad points for more future works - Included three parts: UT + GUI Preference + FGD + + GUI Focus Group User Testing Preference Discussion 9 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

2. Experiment Design 2.2. User Testing - Normal user testing techniques (think-aloud) - Went through 6 sample tasks with 8 participants - Used a hi-fi prototype to simulate our new UI - Controlled experiment with video recording (MORAE) - 1 moderator and multiple observers CONFIDENTIAL CONFIDENTIAL 10 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

2. Experiment Design 2.3. GUI Preference - Asked the same participants to choose our new GUI designs - A sort of multiple-choice survey questions - Asked to choose the best 2 & the worst 1 design CONFIDENTIAL 11 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

2. Experiment Design 2.4. Focus Group Discussion - To collect real users’ preferences and behaviours - In terms of using the current version of Flirtomatic - Focused on gathering additional information - To help redesign the details of our WAP service - Went through 5 topics with 3 groups of users (m, f, & gay) 12 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

3. Good Lessons – Dos and Donts 3.1. Outcomes - Were able to conduct evaluation ourselves – DIY! - Captured some important findings to validate our prototype - Less than 2 wks (Experiment design + experiment + analysis) - Less than 500.00 GBP - Much quicker and cheaper than hiring research firms - Were able to share the importance of usability evaluation with the management level and all other departments - Self-learn about the process of usability evaluation 14 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

3. Good Lessons – Dos and Donts 3.2. Difficulties - Recruitment error: No show up, recruited friends, etc. - MORAE error (due to the inexperienced operation): No video recording, low quality audio recording, etc. - Some mistakes of our colleagues: intruding into the UT room to connect to wireless network, interrupting UT sessions to ask questions to the moderator, giving money to participants during the session, etc. 15 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

3. Good Lessons – Dos and Donts 3.3. Dos - Good preparation in every way: 1. Documentation on task scenario, forms, analysis 2. Train yourself to master any SW tools (like MORAE) 3. Clean up the venue in advance - Share the importance of usability evaluation with everyone 1. With the management level 2. Especially with colleagues nearby the UT room (should politely ask silence during each session) - Share the formal process of evaluation with team members - Educate yourself how to run each session professionally 16 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

3. Good Lessons – Dos and Donts 3.4. Donts - Do not rely on the quick list of recruited users (importance of having some back-up participants) - Do not expect everyone else in office will behave as you do 1. They don’t know how important it is 2. Actually they don’t care what you do - Do not expect your team members understand everything 1. They don’t know how to formally run each session 2. Importance of pre-training for usability evaluation 17 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

3. Good Lessons – Dos and Donts 3.5. Resources - MORAE: http://www.techsmith.com/morae.asp - The Usability Methods Toolbox by James Hom: http://jthom.best.vwh.net/usability - Usability First: http://www.usabilityfirst.com/ - Jakob Nielsen: http://www.useit.com/ 18 Mobile Design UK © 2008 Handmade Mobile. All Rights Reserved.

Cheap& Quick Internal user testing

0 comments

Notes on slide 1

1 Favorite