To Launch or Not to Launch:
How a Good Concept Test could go Bad
One week has passed since Alex decided to employ concept testing to determine if his product idea would be celebrated
by the market or tossed in the bin marked ‘Failure’. Today, he approached his boss, Jane, and explained his work plan.
A big part of his plan was concept testing. He expected that the road ahead would be very smooth: they would get a
research partner on board, test the concept, and find out if the idea would realistically work in the market - or not.
Jane, unfortunately, did not seem as excited about his brilliant solution. In the ten years she’d been in the industry, she’d
seen several such studies unintentionally mislead product managers into getting rid of brilliant ideas and launching
relatively weaker ones in their place. The core issue, as far as Jane was concerned, still remained: To launch or not to
It’s not that Jane doesn’t believe in concept testing. It’s that she knows how easy it is to get important elements of it
wrong, thus leading to inaccurate, easily-misinterpreted results.
After thinking for a moment, she said to Alex:
“I’m not against concept testing, but it’s important to do it right. Before you start testing, make sure you identify and use
the appropriate KPIs and benchmarks for scoring. Let me know if you need help with that.”
How will Alex do this?
Research agencies often focus on incorporating product-relevant KPIs in the research design, for example, test for
likeability, relevance, uniqueness, etc. This allows the researcher to estimate how well the product will fit the market.
However, many times our business objectives aren’t limited to simply launching a product. There are other overarching
business goals that need to be considered. It is critical to reconcile the KPIs to both the product and to these broader
business goals. For example, a concept might test as relevant and unique but fail on the business goal of getting 20%
new people into the category.
If Alex can choose KPIs that speak to both the product and the big picture, this will give him accurate success indicators.
This, in turn, will make the go/no-go decision clearer.
Setting the Broader Business Goals, or Benchmarks, for Accurate Concept Test Scoring
Once the right set of KPIs have been decided upon, another and bigger challenge awaits. How does a Product Manager decide if
the scores coming back from the agency are good, bad, or ugly?
Respondents are often sceptical of new product ideas, especially those that challenge their concept of the status quo or that step
outside their personal comfort zone. In addition, they may not be able to visualize the benefits that a new product will provide.
In Jane’s experience, going by absolute scores in such cases is a good way to kill off promising ideas.
Luckily, there is a way out – Historical data or industry knowledge makes a sound starting point. And for new markets without these
inputs, the other concepts in the study can be used to judge relative performance and customer acceptance. Accurate benchmarks
for the scores help determine if the concept will fare well in market conditions or not.
.Jane, having explained these critical parts of concept testing to Alex, continues:
“You have my full support, Alex – just be sure to choose your KPIs and benchmarks carefully”.
When Alex first considered concept testing his new product, he had no idea it would involve so much thought! Striving not to feel
overwhelmed, he reminds himself: first things first. Find a suitable research partner. One that can also give him the most reliable
and actionable results.
Choosing the Right Key Performance Indicators (KPIs):
Shiva Seth – Manager, Market Research Practice at Absolutdata
Akanksha Mittal – Research Analyst, Market Research Practice at Absolutdata