This document outlines the process and members of a task force formed to evaluate and improve a company's product evaluation process. The task force includes community members, staff from various departments, and is led by the COO/CFO. Over the course of several weeks, the group will brainstorm new directions, choose a lead for each, present concepts, and finalize a new approach. User experience design and testing will then inform the implementation of improvements to better surface the best ideas and make the evaluation process more useful, engaging, and constructive.

1. TASK FORCE:
for the future of social product development
PRODUCT
EVALUATION

2. WHO
• Top community voters / vocal community members:
Matt Fleming, Clinton Fleenor, Judi Sigler, Michelle Brewster, Stacy Prince,
Steven Kramer & Charles Bailey
• John Lott – COO / CFO
• Brian Kerr – UX
• Jessica Marati – Community
• Mitch Lowe- Strategy
• Gaz Brown – Product Design
• Nancy Chen & Anthony Del Plato – Data Analysis
• Nathan Smith & Mike Lacy – Technology

3. WHAT A
SUCCESSFUL
PRODUCT
EVALUATION
PROCESS
LOOKS LIKE:

4. 1. Surfaces the ‘best ideas’ every week.
WHAT A SUCCESSFUL PRODUCT EVALUATION PROCESS DOES:

5. 2. Curation is a fun and engaging
experience for voters and community
members.
WHAT A SUCCESSFUL PRODUCT EVALUATION PROCESS DOES:

6. 3. Data collected is useful in product
design, research, and marketing process.
WHAT A SUCCESSFUL PRODUCT EVALUATION PROCESS DOES:

7. 4. Inventors who do not win walk away
educated, with a ton of insight about
where they fell short.
WHAT A SUCCESSFUL PRODUCT EVALUATION PROCESS DOES:

8. 5. Requires a sustainable amount of
Quirky staff interaction.
WHAT A SUCCESSFUL PRODUCT EVALUATION PROCESS DOES:

9. 6. Rewards all who are involved without
encouraging gaming.
WHAT A SUCCESSFUL PRODUCT EVALUATION PROCESS DOES:

10. 7. Ends in winning ideas that both the
community and Quirky are excited about.
WHAT A SUCCESSFUL PRODUCT EVALUATION PROCESS DOES:

11. 8. Winning ideas enter the design process
with a ton of research, knowledge,
and demand behind them.
WHAT A SUCCESSFUL PRODUCT EVALUATION PROCESS DOES:

12. WHAT
ARE THE
‘BEST IDEAS’?

13. 1. The best ideas solve big problems
(relating to usability, comfort,
and convenience) in a new way.
WHAT ARE THE BEST IDEAS

14. 2. The best ideas represent the
biggest commercial opportunity
for our community & brand.
WHAT ARE THE BEST IDEAS

15. 3. The best ideas are manufacturable.
WHAT ARE THE BEST IDEAS

16. 4. The best ideas don’t go out of
fashion / relevance quickly – they are
around for years (ideally 3+ year lifespan).
WHAT ARE THE BEST IDEAS

17. 5. The best ideas are protectable.
WHAT ARE THE BEST IDEAS

18. 6. The best ideas are often the ones
people ignore or reject.
WHAT ARE THE BEST IDEAS

19. KNOWN
ISSUES:

20. 1. Number of ideas grows each week,
making the curation process more
fatiguing for community members.
KNOWN ISSUES

21. 2. We are, by design, saying ‘get lost’ to all
but two of hundreds of inventors every
week – how do we turn ‘get lost’ into
something constructive and helpful, and
how do we inspire losing inventors to
stick around so they can learn, grow,
and become successful inventors?
KNOWN ISSUES

22. 3. Evaluating an ‘idea’ this early in the
process is risky, because no one (not
even the inventor) fully understands ‘it’ –
because ‘it’ doesn’t exist yet.
“Don’t worry about people stealing your ideas. If they are any good,
you’ll have to ram them down people’s throats”. - Howard Aiken
KNOWN ISSUES

23. 4. Due to relatively low number (about 2000)
active voters and high weight of votes in
defining the top ideas in a given week, the
weight of each inventor’s social graph is
higher then the actual ‘quality’ of an idea.
KNOWN ISSUES

24. 5. People are voting for what they
believe Quirky will like, versus what
they actually like and will buy.
KNOWN ISSUES

25. 6. Seasonality and changes in culture make
things that were ‘no’s’ a short time ago, big
‘yes’s’ shortly thereafter... do we lose
that opportunity, and how do we make
a mineable database of ideas?
KNOWN ISSUES

26. QUESTIONS/
THOUGHT
STARTERS:

27. 1. How do we create a mineable database
of ideas that can somehow come back into
relevance as times change / relevance
changes / lines need expansion, etc?
KNOWN ISSUES

28. 2. How do we avoid a culture (both staff
and community) that gravitates toward
easy wins, but rather encourage a
culture that is drawn toward risk?
KNOWN ISSUES

29. 3. Is a weekly ‘class’ of ideas the best
way to approach evaluation?
other thoughts have included:
• rolling list in each category (ideas must hit a certain threshold
in order to get pushed into staff eval)
• volume-per-category-driven (collect 50 ideas in each
category, then choose one)
• sudden death / elimination round (slowly kill off ideas
throughout the curation process that way we are all focusing
on going deeper into ideas we like)
KNOWN ISSUES

30. 4. How can the community do more
research earlier in the process to make
staff evaluation easier?
KNOWN ISSUES

31. THE
TASKFORCE
PROCESS

32. March 1st- Task Force planning begins
March 4th- Research survey sent to active voters (effort to better understand current climate)
March 10th- Plan complete, Task Force called to arms. Planning deck sent. Basecamp invites
sent to all members. Brainstorm-style discussion begins within Basecamp.
March 15th- Kick-off / brainstorm conference call (full Task Force participation, time TBA).
Quirky-led conversation, resulting in three to five clear directions we can go. Jess will take thorough
notes and post on Basecamp.
A ‘writeboard’ will be started for each of the directions – ‘bullet point / process style’ –
and all Task Force members will work to refine the process, working in a ‘wiki’ environment.
March 24th- A ‘lead’ for each direction is chosen. Lead begins to prepare presentation of
how the process could work.
March 28th- Conference call check-in (full Task Force participation, time TBA).
Each lead will discuss progress / challenges that still exist within their concept –
things they’re worried about, things they’re excited about.
April 7th- Full afternoon @ QHQ. Each ‘lead’ will present their concept to the entire task force,
Ben will join. Final direction will be chosen.
April 8th- UX/UI team briefed on vision / objectives
April 14th- UX / UI team will deliver preliminary wireframes of the new process / post
to Basecamp. Conference Call (full Task Force participation, time TBA)
April 21st- UX/UI refinements
April 27th- Town Meeting presentation of new plan
May- Technology / user testing
June- Implementation of new product evaluation process
THE TASK FORCE PROCESS