Steve Herbst                                           SMDeveloper of Topical Storm                    software-a web-base...
Cartoon by Barbara Smaller,                                                          The New Yorker 4/26/10© 2012   Steve ...
Names don’t work that well compared to product.Google – we worry about the name … will the worldaccept … or Yahoo. Or Lotu...
My product development history      1998                                    2010                                    2011  ...
My product development history      1998                                    2010                                    2011  ...
Daughter’s 6th grade essay topic:                         Warm air causes hurricanes                Using a Google search,...
How doesWarm air                                                warm aircauses                     Rule                   ...
Warm air causes hurricanesNoticing the pattern “A causes B”, the computer asks:   How does warm air cause hurricanes?   Ev...
Warm air causes hurricanesNoticing the pattern “A causes B”, the computer asks:   How does warm air cause hurricanes?   Ev...
Warm air causes hurricanesNoticing the pattern “A causes B”, the computer asks:   How does warm air cause hurricanes?   Ev...
Warm air causes hurricanesNoticing the pattern “A causes B”, the computer asks:   How does warm air cause hurricanes?   Ev...
Too much television causes ADHDNoticing the pattern “A causes B”, the computer asks:   How does too much television cause ...
Too much television causes ADHDNoticing the pattern “A causes B”, the computer asks:   How does too much television cause ...
Too much television causes ADHDNoticing the pattern “A causes B”, the computer asks:   How does too much television cause ...
Product: An outliner that generates questions:   We should focus on market share      What evidence do you have that marke...
My product development history      1998                                    2010                                    2011  ...
2010November Learning (Boston)Conference: Building Learning Communities              © 2012   Steve Herbst Confidential an...
2010High energy, very exciting. All these free tools:                © 2012   Steve Herbst Confidential and Proprietary
2010Lesson learned: Hold steady to the original vision. Don’t rush to change the product in response to trends.Positive ou...
My product development history      1998                                    2010                                    2011  ...
Reality check 2010Showed a civics demo to a history professor friend.     Me: Isn’t this technology super cool? My program...
Reality check 2010Lesson learned: Nobody cares how the software works, they want magic.Positive outcome: user focus Spare ...
My product development history      1998                                    2010                                    2011  ...
Douglas School  Acton MA  May 2011                 © 2012   Steve Herbst Confidential and Proprietary
May 2011        Survey comments from 6th grade writing students:                                         It helps writers ...
May 2011Lessons learned: Sixth graders are awfully perceptive. They don’t want happy cartoon characters. They love the sco...
My product development history      1998                                    2010                                    2011  ...
Software as Show Business                   (SASB)     “I wouldn’t buy this –      even for 5 dollars.      But…”  © 2012 ...
December 2011Early morning realization:          Nobody wants this. © 2012   Steve Herbst Confidential and Proprietary
“Enlightened persons who know                     the True Self work for the                     welfare of the world, una...
Product Use Testing              also called Usability Testinginvolves a task, for example:• make something with a tool• d...
Things to find out:• How quickly people do the task• How accurately• How long they stick with it• How they feel about the ...
MethodSimple:• Watch people use your product on a realistic task• See when they get confused or lose interest• Ask them fo...
Conventional view of testing                                  TEST                                  HERE                  ...
Reality of testingSMALLTESTS…         © 2012   Steve Herbst Confidential and Proprietary
Test early & often• Test a mockup   • For tangible goods: model, prototype   • For software, focus test a “wireframe”:    ...
Testing in large organizations• Professional usability testers• Many test subjects• Fancy equipment – eye tracking, voice ...
thinkeyetracking.comNielsen Norman Group                                                            University of Minnesot...
Simpler non-fanatic approach• A few test subjects• Realistic task• Listen carefully, take notes• Record the session if pos...
Step by step1. Define the goal of this test2. Choose a realistic task3. Select participants4. Prepare participants – expec...
1. Define the goal of this testFind out how to improve the product:     See where people get stuck     Gauge emotional r...
1. Define the goal of this testNote:     A Product Use Test tests the product,        not the participant          Task ...
2. Choose a realistic taskRealistic: Something that people need to do           and that your product needs to facilitateE...
3. Select participantsWho:     People            It takes only five users to uncover 80 percent of            high-level ...
Why do the test subjects know better than us?They know very little about our product       (just like all the people who w...
4. Prepare the participantsSet expectations, give motivation:      Credibility: Who is conducting this test?      How lo...
5. Questionnaire beforehandSample questions:    What’s your profession?    What types of tasks do you do?    (How) do y...
6. Run the testWhile performing the task(s):    Ask them to think aloud:           What they are about to do           ...
7. Ask for reactions, suggestionsPost-interview or questionnaire:    Summarize this experience    What was difficult?   ...
8. Report results to participantsThis is one of the strongest incentives to participate     Get contact info or post on a...
Today’s Product Use Test is different• Everybody all at once• We can’t do think-aloud (unfortunately)• The program will sa...
Today’s Product Use Test1. Log in with your user name & password4. Pretend you want to write a college application   essay...
Upcoming SlideShare
Loading in …5
×

BC MK-803 product use testing 3-15-12

627 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
627
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

BC MK-803 product use testing 3-15-12

  1. 1. Steve Herbst SMDeveloper of Topical Storm software-a web-based outlining program that offerssuggestions to overcome writer’s blockThis presentation for MK-803:My product development story and what Iknow about Product Use Testing © 2012 Steve Herbst Confidential and Proprietary
  2. 2. Cartoon by Barbara Smaller, The New Yorker 4/26/10© 2012 Steve Herbst Confidential and Proprietary
  3. 3. Names don’t work that well compared to product.Google – we worry about the name … will the worldaccept … or Yahoo. Or Lotus. How silly. Lotus. Apple.And yet it’s the product. The product really, reallymatters. -Dan Bricklin, co-founder of VisiCalc -Interview May 7, 2004 © 2012 Steve Herbst Confidential and Proprietary
  4. 4. My product development history 1998 2010 2011 Groundwork: Stay on target: Validation: NEXTReasoning software Education market Field test at STEPS runs on a PC goes crazy for Douglas School collaboration tools In Acton MA 2006 Inspiration: 2010 2011 Daughter writes Reality check: Enlightenment: crappy essay Customers Software is expect perfection Show biz © 2012 Steve Herbst Confidential and Proprietary
  5. 5. My product development history 1998 2010 2011 Groundwork: Stay on target: Validation: NEXTReasoning software Education market Field test at STEPS runs on a PC goes crazy for Douglas School collaboration tools In Acton MA 2006 Inspiration: 2010 2011 Daughter writes Reality check: Enlightenment: crappy essay Customers Software is expect perfection Show biz © 2012 Steve Herbst Confidential and Proprietary
  6. 6. Daughter’s 6th grade essay topic: Warm air causes hurricanes Using a Google search, she finds:Hurricanes start when warm, moist air from the ocean surface begins to rise rapidly, where itencounters cooler air that causes the warm water vapor to condense and to form storm clouds anddrops of rain. The condensation also releases latent heat, which warms the cool air above, causing itto rise and make way for more warm humid air from the ocean below.As this cycle continues, more warm moist air is drawn into the developing storm and more heat istransferred from the surface of the ocean to the atmosphere. This continuing heat exchange creates awind pattern that spirals around a relatively calm center, or eye, like water swirling down a drain.Done! © 2012 2011 Steve Herbst Confidential and Proprietary
  7. 7. How doesWarm air warm aircauses Rule causehurricanes hurricanes ? © 2012 Steve Herbst Confidential and Proprietary
  8. 8. Warm air causes hurricanesNoticing the pattern “A causes B”, the computer asks: How does warm air cause hurricanes? Evidence that warm air causes hurricanes What causes warm air? What do hurricanes cause? What else causes hurricanes? What else does warm air cause? Can warm air be used to predict hurricanes? Can preventing warm air prevent hurricanes? Do different kinds of warm air cause different kinds of hurricanes?And so on. © 2012 Steve Herbst Confidential and Proprietary
  9. 9. Warm air causes hurricanesNoticing the pattern “A causes B”, the computer asks: How does warm air cause hurricanes? Evidence that warm air causes hurricanes What causes warm air? What do hurricanes cause? What else causes hurricanes? What else does warm air cause? Can warm air be used to predict hurricanes? Can preventing warm air prevent hurricanes? Do different kinds of warm air cause different kinds of hurricanes?And so on. © 2012 Steve Herbst Confidential and Proprietary
  10. 10. Warm air causes hurricanesNoticing the pattern “A causes B”, the computer asks: How does warm air cause hurricanes? Evidence that warm air causes hurricanes What causes warm air? What do hurricanes cause? What else causes hurricanes? What else does warm air cause? Can warm air be used to predict hurricanes? Can preventing warm air prevent hurricanes? Do different kinds of warm air cause different kinds of hurricanes?And so on. © 2012 Steve Herbst Confidential and Proprietary
  11. 11. Warm air causes hurricanesNoticing the pattern “A causes B”, the computer asks: How does warm air cause hurricanes? Evidence that warm air causes hurricanes What causes warm air? What do hurricanes cause? What else causes hurricanes? What else does warm air cause? Can warm air be used to predict hurricanes? Can preventing warm air prevent hurricanes? Do different kinds of warm air cause different kinds of hurricanes?And so on. © 2012 Steve Herbst Confidential and Proprietary
  12. 12. Too much television causes ADHDNoticing the pattern “A causes B”, the computer asks: How does too much television cause ADHD? Evidence that too much television causes ADHD What causes too much television? What does ADHD cause? What else causes ADHD? What else does too much television cause? Can too much television be used to predict ADHD? Can preventing too much television prevent ADHD? Do different kinds of too much television cause different kids of ADHD?And so on. © 2012 Steve Herbst Confidential and Proprietary
  13. 13. Too much television causes ADHDNoticing the pattern “A causes B”, the computer asks: How does too much television cause ADHD? Evidence that too much television causes ADHD What causes too much television? What does ADHD cause? What else causes ADHD? What else does too much television cause? Can too much television be used to predict ADHD? Can preventing too much television prevent ADHD? Do different kinds of too much television cause different kids of ADHD?And so on. © 2012 Steve Herbst Confidential and Proprietary
  14. 14. Too much television causes ADHDNoticing the pattern “A causes B”, the computer asks: How does too much television cause ADHD? Evidence that too much television causes ADHD What causes too much television? What does ADHD cause? What else causes ADHD? What else does too much television cause? Can too much television be used to predict ADHD? Can preventing too much television prevent ADHD? Do different kinds of too much television cause different kids of ADHD?And so on. © 2012 Steve Herbst Confidential and Proprietary
  15. 15. Product: An outliner that generates questions: We should focus on market share What evidence do you have that market share is important? What other factors are also important? Are there any unspoken assumptions behind this statement? What steps can we take to focus on market share? What is the purpose of market share? Who knows the most about market share? Who might disagree with this statement? How can we research this recommendation further? © 2012 Steve Herbst Confidential and Proprietary
  16. 16. My product development history 1998 2010 2011 Groundwork: Stay on target: Validation: NEXTReasoning software Education market Field test at STEPS runs on a PC goes crazy for Douglas School collaboration tools In Acton MA 2006 Inspiration: 2010 2011 Daughter writes Reality check: Enlightenment: crappy essay Customers Software is expect perfection Show biz © 2012 Steve Herbst Confidential and Proprietary
  17. 17. 2010November Learning (Boston)Conference: Building Learning Communities © 2012 Steve Herbst Confidential and Proprietary
  18. 18. 2010High energy, very exciting. All these free tools: © 2012 Steve Herbst Confidential and Proprietary
  19. 19. 2010Lesson learned: Hold steady to the original vision. Don’t rush to change the product in response to trends.Positive outcome: clarity Plenty of existing tools for sharing content. But there may be value in helping people create good written content to share. © 2012 Steve Herbst Confidential and Proprietary
  20. 20. My product development history 1998 2010 2011 Groundwork: Stay on target: Validation: NEXTReasoning software Education market Field test at STEPS runs on a PC goes crazy for Douglas School collaboration tools In Acton MA 2006 Inspiration: 2010 2011 Daughter writes Reality check: Enlightenment: crappy essay Customers Software is expect perfection Show biz © 2012 Steve Herbst Confidential and Proprietary
  21. 21. Reality check 2010Showed a civics demo to a history professor friend. Me: Isn’t this technology super cool? My program generates all these ideas with just a few powerful rules! John: This is bad grammar. And this question is vague. Teachers won’t accept it. © 2012 Steve Herbst Confidential and Proprietary
  22. 22. Reality check 2010Lesson learned: Nobody cares how the software works, they want magic.Positive outcome: user focus Spare us the fancy “Artificial Intelligence” technology. Make it simple; hard-code the answers. Don’t mention “rules”. Concentrate on what people want from the product. © 2012 Steve Herbst Confidential and Proprietary
  23. 23. My product development history 1998 2010 2011 Groundwork: Stay on target: Validation: NEXTReasoning software Education market Field test at STEPS runs on a PC goes crazy for Douglas School collaboration tools In Acton MA 2006 Inspiration: 2010 2011 Daughter writes Reality check: Enlightenment: crappy essay Customers Software is expect perfection Show biz © 2012 Steve Herbst Confidential and Proprietary
  24. 24. Douglas School Acton MA May 2011 © 2012 Steve Herbst Confidential and Proprietary
  25. 25. May 2011 Survey comments from 6th grade writing students: It helps writers that I would use this Yes I think it is a very have writers block program again successful website. and are unable to because it comes Could be a great tool for organize there ideas up with ideas that i kids writing essays. while trying to write would have not there essay or story. come up with.i really do enjoy filling inall of the checklists things I would definetly use a I would use this weband answering the program like this site agen becausequestions. i really think again because it really never befor have ithat topical storm is a makes you think about been able to make avery productive and fun what you are writing first draft of an essayway to write an essay. about. in a single day before. Thank you! © 2012 Steve Herbst Confidential and Proprietary
  26. 26. May 2011Lessons learned: Sixth graders are awfully perceptive. They don’t want happy cartoon characters. They love the scoreboard. Make it Smartboard ready, touch-based. Simplify it beyond belief. Getting people to use the product is priceless. © 2012 Steve Herbst Confidential and Proprietary
  27. 27. My product development history 1998 2010 2011 Groundwork: Stay on target: Validation: NEXTReasoning software Education market Field test at STEPS runs on a PC goes crazy for Douglas School collaboration tools In Acton MA 2006 Inspiration: 2010 2011 Daughter writes Reality check: Enlightenment: crappy essay Customers Software is expect perfection Show biz © 2012 Steve Herbst Confidential and Proprietary
  28. 28. Software as Show Business (SASB) “I wouldn’t buy this – even for 5 dollars. But…” © 2012 Steve Herbst Confidential and Proprietary
  29. 29. December 2011Early morning realization: Nobody wants this. © 2012 Steve Herbst Confidential and Proprietary
  30. 30. “Enlightened persons who know the True Self work for the welfare of the world, unattached, ever helping to point humanity toward dharma (right action, living a truth-based life).” - THE BHAGAVAD GITA, 200 BCE – 200 AD© 2012 Steve Herbst Confidential and Proprietary
  31. 31. Product Use Testing also called Usability Testinginvolves a task, for example:• make something with a tool• drive a vehicle• fill out a form• chop carrots• sleep in a tent © 2012 Steve Herbst Confidential and Proprietary
  32. 32. Things to find out:• How quickly people do the task• How accurately• How long they stick with it• How they feel about the experience afterwards • Would they buy? Recommend?• How much they remember later © 2012 Steve Herbst Confidential and Proprietary
  33. 33. MethodSimple:• Watch people use your product on a realistic task• See when they get confused or lose interest• Ask them for suggestions afterwards• Use this priceless information to improve the product• Testing is useful at every stage of development © 2012 Steve Herbst Confidential and Proprietary
  34. 34. Conventional view of testing TEST HERE FIX © 2012 Steve Herbst Confidential and Proprietary
  35. 35. Reality of testingSMALLTESTS… © 2012 Steve Herbst Confidential and Proprietary
  36. 36. Test early & often• Test a mockup • For tangible goods: model, prototype • For software, focus test a “wireframe”: © 2012 Steve Herbst Confidential and Proprietary
  37. 37. Testing in large organizations• Professional usability testers• Many test subjects• Fancy equipment – eye tracking, voice stress• Statistical analysis of results• Feh © 2012 Steve Herbst Confidential and Proprietary
  38. 38. thinkeyetracking.comNielsen Norman Group University of Minnesota © 2012 Steve Herbst Confidential and Proprietary
  39. 39. Simpler non-fanatic approach• A few test subjects• Realistic task• Listen carefully, take notes• Record the session if possible• Think, fix See: Steve Krug, Rocket Surgery Made Easy © 2012 Steve Herbst Confidential and Proprietary
  40. 40. Step by step1. Define the goal of this test2. Choose a realistic task3. Select participants4. Prepare participants – expectation, motivation5. Pre-questionnaire? Understand the person6. Run the test7. Ask for reactions, suggestions8. Report results to the participants © 2012 Steve Herbst Confidential and Proprietary
  41. 41. 1. Define the goal of this testFind out how to improve the product:  See where people get stuck  Gauge emotional reactions  Discover the most & least attractive features  Compare several products or concepts (“Use each of these machines to make espresso”) © 2012 Steve Herbst Confidential and Proprietary
  42. 42. 1. Define the goal of this testNote:  A Product Use Test tests the product, not the participant  Task should not be overly difficult  Everything the user does is right; the product is wrong © 2012 Steve Herbst Confidential and Proprietary
  43. 43. 2. Choose a realistic taskRealistic: Something that people need to do and that your product needs to facilitateExamples of tasks for a website:  Create a document  Make a purchase  Find the answer to a question  Sign up (how enticing is the lead-up?) © 2012 Steve Herbst Confidential and Proprietary
  44. 44. 3. Select participantsWho:  People It takes only five users to uncover 80 percent of high-level usability problems. -Jakob Nielsen © 2012 Steve Herbst Confidential and Proprietary
  45. 45. Why do the test subjects know better than us?They know very little about our product (just like all the people who will use it someday)They’re detached They’re focused on their own needs and couldn’t care less how much more work we have to do.They’re in a hurry (just like all the people …) © 2012 Steve Herbst Confidential and Proprietary
  46. 46. 4. Prepare the participantsSet expectations, give motivation:  Credibility: Who is conducting this test?  How long will the test take?  If you will be recording, tell them  Guarantee confidentiality  Promise a reward: cash, iTunes card, free product or subscription  Ask them to sign a consent form  Let them view the test results later! © 2012 Steve Herbst Confidential and Proprietary
  47. 47. 5. Questionnaire beforehandSample questions:  What’s your profession?  What types of tasks do you do?  (How) do you currently use this type of product?  What difficulties or needs do you have?  What other similar products do you know?  Blue sky: What product would you love to have? © 2012 Steve Herbst Confidential and Proprietary
  48. 48. 6. Run the testWhile performing the task(s):  Ask them to think aloud:  What they are about to do  How they plan to do it  What step they are trying now  What they think a feature does  What surprises them  What questions they have  Why they want to kill you © 2012 Steve Herbst Confidential and Proprietary
  49. 49. 7. Ask for reactions, suggestionsPost-interview or questionnaire:  Summarize this experience  What was difficult?  What was confusing?  What would you change?  You know, stuff like that © 2012 Steve Herbst Confidential and Proprietary
  50. 50. 8. Report results to participantsThis is one of the strongest incentives to participate  Get contact info or post on a website  Report what most people liked & hated  Report any interesting ideas that came up © 2012 Steve Herbst Confidential and Proprietary
  51. 51. Today’s Product Use Test is different• Everybody all at once• We can’t do think-aloud (unfortunately)• The program will save your work and comments, and I will look at them later• We’ll discuss when done• I will post a summary of the results © 2012 Steve Herbst Confidential and Proprietary
  52. 52. Today’s Product Use Test1. Log in with your user name & password4. Pretend you want to write a college application essay5. Start by knowing nothing about the product3. See how far you get (that’s the goal of this test)4. Click this button to enter your comments:5. Whole class discussion at 9PMResults will be posted at: www.topicalstorm.com/BC/results This PowerPoint at: www.topicalstorm.com/BC/slides My email: steve.herbst@comcast.net © 2012 Steve Herbst Confidential and Proprietary

×