Recommended
Recommended
More Related Content
Similar to When Crowds Give You Lemons: Filtering Innovative Ideas using a Diverse-Bag-of-Lemons Strategy
Similar to When Crowds Give You Lemons: Filtering Innovative Ideas using a Diverse-Bag-of-Lemons Strategy (20)
Recently uploaded
Recently uploaded (20)
When Crowds Give You Lemons: Filtering Innovative Ideas using a Diverse-Bag-of-Lemons Strategy
- 1. When Crowds Give You Lemons: Filtering Innovative Ideas using a Diverse-Bag-of- Lemons Strategy Ioanna Lykourentzou, Faez Ahmed, Costas Papastathis, Irwyn Sadien, Konstantinos Papangelis
- 3. How to filter the most innovative ideas involving the crowd? • Example: Redesigning the Bus Experience • 36 ideas • Symbols indicate the broad theme of each idea
- 4. Majority voting • Each voter can upvote any number of ideas
- 5. Re-ranking in majority voting • Ideas are ranked in descending order based on the number of upvotes they receive
- 6. Snowball effect • Ideas which received the first initial upvotes are more likely to be seen by subsequent users
- 7. Problems with majority voting • Prone to quickly locking into a fairly static ranking due to positive feedback loops • Crowds are less effective in distinguishing mediocre from excellent ideas • Prone to quickly locking into a fairly static ranking due to positive feedback loops • Crowds are less effective in distinguishing mediocre from excellent ideas
- 8. Inspiration: Bag of lemons • Bag of Lemons: Ask voters to identify the worst rather than the best ideas using a multi-voting approach (Klein and Garcia (2015))
- 9. DBLemons: Diverse Bag of Lemons We propose a crowd-voting method, which overcomes these problems by using Bag of Lemons voting and exposes voters to a wider idea spectrum, thanks to a dynamic diversity-based ranking system Algorithm implemented in three steps: 1. Identify clusters to which each idea belongs 2. Rank order ideas using our algorithm 3. Ask crowd worker to vote using lemons
- 10. DBLemons Trade-off Function • Utility function is submodular, hence a greedy algorithm is theoretically guaranteed to provide the best possible polynomial-time approximation to the optimal solution.
- 12. DBLemon visual example • Identify clusters • Transportation cluster
- 13. DBLemon visual example • Identify clusters • Computer and Technology cluster
- 14. DBLemon visual example • Clusters are equally represented
- 15. DBLemon visual example • First voter distributes 5 lemons
- 16. DBLemon visual example • Ideas sorted by quality and diversity. • Ideas with more lemons are pushed down
- 17. DBLemon visual example • Second voter distributes 5 lemons
- 18. DBLemon visual example • Ideas with least lemons are considered highest quality
- 19. Women Safety Ideation Contest • The dataset we worked with • How might we make low-income urban areas safer and more empowering for women and girls?
- 20. Process • We created 52 idea set summaries of existing OpenIDEO ideas. • Each summary was reviewed sequentially by 3 reviewers. Each idea evaluated by 30 crowd workers on 5-point Likert scale. • Top 30% selected as golden set • Comparison between majority voting, Bag of Lemons (BOL), and DBLemons approaches
- 21. Testing platform
- 22. Results • DBLemons is more accurate, less time- consuming, and reduces the idea space in half while still retaining 94% of the top ideas. Performance comparison of the three ranking strategies. The dashed line shows the golden set cardinality cut-off
- 23. Results • Progressive ranking per strategy, descending quality order. All strategies improve with the number of voters, but DBLemons does so faster
- 24. Results • DBLemons provides a more even coverage of all idea clusters • Entropy of idea clusters - first page
- 25. Results • People made multiple hops to compare ideas with each other but took less time overall
- 26. Contributions We show that: • Majority voting is more time consuming and less accurate than the other two alternatives • Balancing idea quality and idea representativeness improves the filtering of high-quality ideas • Bag of Lemons with dynamic ranking has similar filtering efficiency as Majority voting, but finds good ideas faster. This result confirms the promise of the technique proposed by Klein and Garcia on the dynamic setting
- 27. Future work • The methods developed have wide ranging impact on large- and small- scale ideation contests. • Few directions of future research are: • Systematically study how to select number of lemons • Study scalability challenges for very large idea datasets