BC MK-803 product use testing 3-15-12

Steve Herbst
SM
Developer of Topical Storm software
-a web-based outlining program that offers
suggestions to overcome writer’s block

This presentation for MK-803:
My product development story and what I
know about Product Use Testing

© 2012 Steve Herbst Confidential and Proprietary

Cartoon by Barbara Smaller,
The New Yorker 4/26/10


Names don’t work that well compared to product.
Google – we worry about the name … will the world
accept … or Yahoo. Or Lotus. How silly. Lotus. Apple.
And yet it’s the product. The product really, really
matters.
-Dan Bricklin, co-founder of VisiCalc
-Interview May 7, 2004


My product development history

1998 2010 2011
Groundwork: Stay on target: Validation: NEXT
Reasoning software Education market Field test at STEPS
runs on a PC goes crazy for Douglas School
collaboration tools In Acton MA

2006
Inspiration: 2010 2011
Daughter writes Reality check: Enlightenment:
crappy essay Customers Software is
expect perfection Show biz


Daughter’s 6th grade essay topic:

Warm air causes hurricanes
Using a Google search, she finds:

Hurricanes start when warm, moist air from the ocean surface begins to rise rapidly, where it
encounters cooler air that causes the warm water vapor to condense and to form storm clouds and
drops of rain. The condensation also releases latent heat, which warms the cool air above, causing it
to rise and make way for more warm humid air from the ocean below.

As this cycle continues, more warm moist air is drawn into the developing storm and more heat is
transferred from the surface of the ocean to the atmosphere. This continuing heat exchange creates a
wind pattern that spirals around a relatively calm center, or eye, like water swirling down a drain.

Done!

© 2012
2011 Steve Herbst Confidential and Proprietary

How does
Warm air warm air
causes Rule cause
hurricanes hurricanes
?



Noticing the pattern “A causes B”, the computer asks:

How does warm air cause hurricanes?
Evidence that warm air causes hurricanes
What causes warm air?
What do hurricanes cause?
What else causes hurricanes?
What else does warm air cause?
Can warm air be used to predict hurricanes?
Can preventing warm air prevent hurricanes?
Do different kinds of warm air cause different kinds of hurricanes?
And so on.





And so on.


Too much television causes ADHD


How does too much television cause ADHD?
Evidence that too much television causes ADHD
What causes too much television?
What does ADHD cause?
What else causes ADHD?
What else does too much television cause?
Can too much television be used to predict ADHD?
Can preventing too much television prevent ADHD?
Do different kinds of too much television cause different kids of ADHD?
And so on.





And so on.


Product: An outliner that generates questions:

We should focus on market share
What evidence do you have that market share is important?
What other factors are also important?
Are there any unspoken assumptions behind this statement?
What steps can we take to focus on market share?
What is the purpose of market share?
Who knows the most about market share?
Who might disagree with this statement?
How can we research this recommendation further?


2010

November Learning (Boston)
Conference: Building Learning Communities


2010

High energy, very exciting. All these free tools:


2010

Lesson learned:
Hold steady to the original vision. Don’t rush to change the
product in response to trends.

Positive outcome: clarity
Plenty of existing tools for sharing content. But there may be
value in helping people create good written content to share.


Reality check 2010
Showed a civics demo to a history professor friend.

Me: Isn’t this technology super cool? My program
generates all these ideas with just a few powerful
rules!

John: This is bad grammar. And this question is
vague. Teachers won’t accept it.


Reality check 2010

Lesson learned:
Nobody cares how the software works, they want magic.

Positive outcome: user focus
Spare us the fancy “Artificial Intelligence” technology.
Make it simple; hard-code the answers. Don’t mention “rules”.
Concentrate on what people want from the product.


Douglas School
Acton MA
May 2011


May 2011

Survey comments from 6th grade writing students:

It helps writers that I would use this
Yes I think it is a very have writers block program again
successful website. and are unable to because it comes
Could be a great tool for organize there ideas up with ideas that i
kids writing essays. while trying to write would have not
there essay or story. come up with.

i really do enjoy filling in
all of the checklists things I would definetly use a I would use this web
and answering the program like this site agen because
questions. i really think again because it really never befor have i
that topical storm is a makes you think about been able to make a
very productive and fun what you are writing first draft of an essay
way to write an essay. about. in a single day before.
Thank you!


May 2011

Lessons learned:
Sixth graders are awfully perceptive.
They don’t want happy cartoon characters.
They love the scoreboard.
Make it Smartboard ready, touch-based.
Simplify it beyond belief.

Getting people to use the product is priceless.

Software as Show Business
(SASB)

“I wouldn’t buy this –
even for 5 dollars.
But…”


December 2011

Early morning realization:
Nobody wants this.


“Enlightened persons who know
the True Self work for the
welfare of the world, unattached,
ever helping to point humanity
toward dharma (right action,
living a truth-based life).”

- THE BHAGAVAD GITA, 200 BCE – 200 AD


Product Use Testing
also called Usability Testing

involves a task, for example:
• make something with a tool
• drive a vehicle
• fill out a form
• chop carrots
• sleep in a tent


Things to find out:

• How quickly people do the task
• How accurately
• How long they stick with it
• How they feel about the experience afterwards
• Would they buy? Recommend?
• How much they remember later


Method
Simple:
• Watch people use your product on a realistic task
• See when they get confused or lose interest
• Ask them for suggestions afterwards
• Use this priceless information to improve the product
• Testing is useful at every stage of development


Conventional view of testing

TEST
HERE

FIX


Reality of testing

SMALL
TESTS…


Test early & often

• Test a mockup
• For tangible goods: model, prototype
• For software, focus test a “wireframe”:


Testing in large organizations

• Professional usability testers
• Many test subjects
• Fancy equipment – eye tracking, voice stress
• Statistical analysis of results
• Feh


thinkeyetracking.com

Nielsen Norman Group University of Minnesota

Simpler non-fanatic approach

• A few test subjects
• Realistic task
• Listen carefully, take notes
• Record the session if possible
• Think, fix

See: Steve Krug,
Rocket Surgery Made Easy


Step by step

1. Define the goal of this test
2. Choose a realistic task
3. Select participants
4. Prepare participants – expectation, motivation
5. Pre-questionnaire? Understand the person
6. Run the test
7. Ask for reactions, suggestions
8. Report results to the participants



Find out how to improve the product:
 See where people get stuck
 Gauge emotional reactions
 Discover the most & least attractive features
 Compare several products or concepts
(“Use each of these machines to make espresso”)



Note:

 A Product Use Test tests the product,
not the participant
 Task should not be overly difficult

 Everything the user does is right; the
product is wrong


2. Choose a realistic task

Realistic: Something that people need to do
and that your product needs to facilitate

Examples of tasks for a website:
 Create a document
 Make a purchase
 Find the answer to a question
 Sign up (how enticing is the lead-up?)


3. Select participants

Who:
 People

It takes only five users to uncover 80 percent of
high-level usability problems.
-Jakob Nielsen


Why do the test subjects know better than us?

They know very little about our product
(just like all the people who will use it someday)

They’re detached
They’re focused on their own needs and couldn’t
care less how much more work we have to do.

They’re in a hurry
(just like all the people …)


4. Prepare the participants
Set expectations, give motivation:
 Credibility: Who is conducting this test?
 How long will the test take?
 If you will be recording, tell them
 Guarantee confidentiality
 Promise a reward: cash, iTunes card, free
product or subscription
 Ask them to sign a consent form
 Let them view the test results later!


5. Questionnaire beforehand

Sample questions:
 What’s your profession?
 What types of tasks do you do?
 (How) do you currently use this type of product?
 What difficulties or needs do you have?
 What other similar products do you know?
 Blue sky: What product would you love to have?


6. Run the test

While performing the task(s):
 Ask them to think aloud:
 What they are about to do
 How they plan to do it
 What step they are trying now
 What they think a feature does
 What surprises them
 What questions they have
 Why they want to kill you


7. Ask for reactions, suggestions

Post-interview or questionnaire:
 Summarize this experience
 What was difficult?
 What was confusing?
 What would you change?
 You know, stuff like that


8. Report results to participants

This is one of the strongest incentives to participate
 Get contact info or post on a website
 Report what most people liked & hated
 Report any interesting ideas that came up


Today’s Product Use Test is different

• Everybody all at once
• We can’t do think-aloud (unfortunately)
• The program will save your work and comments,
and I will look at them later
• We’ll discuss when done
• I will post a summary of the results


Today’s Product Use Test

1. Log in with your user name & password
4. Pretend you want to write a college application
essay
5. Start by knowing nothing about the product
3. See how far you get (that’s the goal of this test)
4. Click this button to enter your comments:
5. Whole class discussion at 9PM
Results will be posted at: www.topicalstorm.com/BC/results
This PowerPoint at: www.topicalstorm.com/BC/slides
My email: steve.herbst@comcast.net


BC MK-803 product use testing 3-15-12

Recommended

Recommended

More Related Content

Similar to BC MK-803 product use testing 3-15-12

Similar to BC MK-803 product use testing 3-15-12 (20)

Recently uploaded

Recently uploaded (20)

BC MK-803 product use testing 3-15-12