Accuracy of Small-Group Estimation


Published on

This is a presentation about a paper that was accepted into Cognitive Science Society, August 2010. Co-authored with Dr. Michael D. Lee, Full Professor at UC Irvine.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Ask three players. Calculate average then compare.
  • When looking at individual estimates, they are $9.45 away from the true price. When aggregating the entire groups’ estimates, the outcome is $7.52 from the true price. Preliminary analysis.
  • Focus on small groups because woc can be clearly seen in groups of threes. And because we can’t have too many people competing each other. Too confusing for people, too confusing for analyses as well.
  • For the competitive condition, we implemented a cognitive model as an analysis of the estimate.
  • Cooperative group average not so different with consensus.
  • WoC quick stabilization but still better than small group analyses. Competing participants possibly feel discouraged to anchor and mimic other bids. Forced to incorporate individual pov and be independent!
  • Start of with two trials of the estimation task. Ask random person.
  • Optimal dm with 3 players, maximum bid of 50. Dotted line represents probability of winning if that player makes that bid. Solid line is the probability if they will make that choice. Gaussian true price distribution. This example shows that each player maximizes their chances of winning by using certain game strategies.
  • Accuracy of Small-Group Estimation

    1. 1. Accuracy of Small-Group Estimation and the Wisdom of Crowds Jenny Shi Michael D. Lee Department of Cognitive Science University of California, Irvine
    2. 2. Cheese tray knife set Crafted of sustainable bamboo and sleek stainless steel, these five distinct blades carve up your brie and Roquefort with ease. Price in Dollars ($1-$50): Real price: $38
    3. 3. Experimental Stimuli <ul><li>100 everyday items </li></ul><ul><ul><li>Images, description, prices </li></ul></ul><ul><ul><li>Between $5-$45 </li></ul></ul><ul><ul><li>Obtained through shopping websites </li></ul></ul><ul><li>Two sets of 50 </li></ul><ul><ul><li>Items uniformly distributed by price </li></ul></ul><ul><ul><li>Each item = 1 trial </li></ul></ul>
    4. 4. Wisdom of Crowds <ul><li>Groups of people can be smarter than the best individuals among them in the right conditions (Surowiecki, 2005) </li></ul><ul><li>A crowd can be “wise” when four conditions are met: </li></ul><ul><ul><li>Diversity : Each individual has own unique view </li></ul></ul><ul><ul><li>Independence : Less relying on others </li></ul></ul><ul><ul><li>Decentralization : Draw info from different sources </li></ul></ul><ul><ul><li>Aggregation : Turn individual to collective decision </li></ul></ul>
    5. 5. Preliminary Analysis: Individual vs. Crowds <ul><li>Examined the mean average deviation of the price estimations of 22 participants. </li></ul><ul><li>Looking at each individual serves a lower bound </li></ul><ul><li>Standard Wisdom of Crowd analysis serves as upper bound. </li></ul>
    6. 6. Current Study <ul><li>What if there are only small groups available? </li></ul><ul><ul><li>groups of three individuals </li></ul></ul><ul><ul><li>Between subjects design </li></ul></ul><ul><ul><li>Priming, cooperative and competitive settings </li></ul></ul><ul><li>Research questions: </li></ul><ul><ul><li>Which of these settings lead to better or worse estimation of the true prices? </li></ul></ul><ul><ul><li>How does the best setting compare to the individuals and standard wisdom of crowd analysis (our preliminary analyses)? </li></ul></ul>
    7. 7. Experimental Conditions <ul><li>Condition type </li></ul><ul><ul><li>With two primes (drawn from previous data sets) </li></ul></ul><ul><ul><li>Cooperate by hearing each other’s bids </li></ul></ul><ul><ul><li>Cooperate by agreeing on an estimate </li></ul></ul><ul><ul><li>Compete with each other by playing the Price is Right game </li></ul></ul><ul><li>Participants cooperating or competing with each other estimated sequentially and systematically alternated between each trial. </li></ul>
    8. 8. The Price is Right <ul><li>Rules: To win the game, player must bid closest to the retail price without going over. </li></ul><ul><li>Players can bid as high as they want, but they cannot bid the same amount as others or bid less than $1. </li></ul>
    9. 9. Price is Right encourages strategic estimation Item for bid: Ipod $150 $165 $1
    10. 10. Cognitive model for competition estimate <ul><li>(Lee & Shi, 2010) </li></ul><ul><li>Bottom line: Instead of aggregating the “raw” estimates from participants that competed, we used a cognitive model to infer their latent knowledge. </li></ul>wx (a,b,c, μ,σ ) π c (c | a,b,μ, σ )= w3 (a b,c,μ, σ ) p ( μ,σ | a,b,c) p (a,b,c | μ,σ ) p ( μ,σ ) … blah blah blah.
    11. 11. Results for small group estimates $9.36 $8.82 $8.79
    12. 12. Competitive Results <ul><li>Competitive MAD: $8.05 </li></ul><ul><li>Primed MAD: $9.36 </li></ul><ul><li>Cooperative Average MAD: $8.82 </li></ul><ul><li>Cooperative Consensus MAD: $8.79 </li></ul><ul><li>Competitive estimate was better than both primed and cooperative </li></ul>
    13. 13. Summary of our results <ul><li>Wisdom of crowds performs best </li></ul><ul><ul><li>Four conditions were present </li></ul></ul><ul><li>Competitive outperformed both cooperative and priming. </li></ul><ul><ul><li>Competing participants discouraged to mimic other bids because of winning incentive. </li></ul></ul><ul><ul><li>Participants that cooperate or were primed may be dependent on other participants in the group or additional information given. </li></ul></ul>
    14. 14. Conclusion <ul><li>Wisdom of crowd analysis is superior to any other aggregation method. </li></ul><ul><ul><li>Resourceful in extracting information from people. </li></ul></ul><ul><li>Competition > Cooperation > Individual </li></ul><ul><ul><li>Groups perform better than individuals in estimation tasks. </li></ul></ul><ul><ul><li>Cooperation worse than competition possibly because lack of independence. </li></ul></ul><ul><li>Using cognitive models is an efficient way of combining knowledge across individuals. </li></ul><ul><ul><li>Helps us understand both the observed behavior and the reasoning behind it. </li></ul></ul>
    15. 15. <ul><li>Thanks! </li></ul>
    16. 16. References <ul><li>Lee, M.D., & Shi, J. (2010).  The accuracy of small-group estimation and the wisdom of crowds. In R. Catrambone, & S. Ohlsson (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society . Austin, TX: Cognitive Science Society. </li></ul><ul><li>Surowiecki, J. (2004). The wisdom of crowds. New York: Random House. </li></ul>
    17. 17. Extra Slides
    18. 18. Starbucks stainless steel tumbler Enjoy your favorite Starbucks brew in this 10-oz. stainless steel tumbler bottle with convenient handles. Price in Dollars ($1-$50): Real price: $34
    19. 19. Optimal Price is Right Bidding <ul><li>For just 3 players, bidding between $1-$50 </li></ul>
    20. 20. Toaster Inference <ul><li>Participants were shown a $28 toaster </li></ul><ul><ul><li>Bid $31, $28, $1 </li></ul></ul><ul><ul><ul><li>Mean of data is $20 </li></ul></ul></ul><ul><ul><ul><li>Mean of inferred latent price distribution is $29 </li></ul></ul></ul>