Michael S. Bernstein, Mark S. Ackerman, Ed H. Chi, Robert C. Miller<br />
Michael S. Bernstein, Mark S. Ackerman, Ed H. Chi, Robert C. Miller<br />
Michael S. Bernstein, Mark S. Ackerman, Ed H. Chi, Robert C. Miller<br />
Ratio of “Understanding Users” papers<br />to “Systems, Tools, Architectures and Infrastructure” papers <br />submitted to...
Trouble: Exponential Growth<br />Your usage data is not really compelling because only a small fraction of Facebook is usi...
Suggestion: Exponential Growth<br />Separate evaluation of spread from steady-state.<br />Which claim is the paper making?...
Trouble: Snowball Sampling<br />The authors’ choice of study method – snowball sampling their system by advertising within...
Suggestion: Snowball Sampling<br />Snowballing is inevitablein social systems. It is fundamental to how they operate.<br />
NoveltyBetween a Rock and a Hard Science<br />
sociotechnical<br />
sociotechnical<br />studiersbuilders<br />
sociotechnical<br />studiersbuilders<br />Fatal Flaw Fallacy [Olsen]<br />Ecological validity at the cost of internal vali...
sociotechnical<br />studiersbuilders<br />Show us elegant complexity.(simple ideas that enable complex scenarios)<br />We ...
socio  technical<br />studiers  builders<br />Build a technically interesting system <br />    (that is hard to spread or ...
The contribution needs to take one strong stance or another. Either it describes a novel system or a novel social interact...
Create a shared understanding<br />of research contributions<br />
social technical<br />New forms of social interaction<br />	Shared organizational memory [Ackerman 1994]<br />Designs that...
social technical<br />Designs collecting or powered by social data<br />Wikidashboard [Suh et al. 2008]; sense.us [Heeret ...
social technical<br />×<br />Paired contributions can increase each others’ value<br />ManyEyes<br />[Viégas et al. 2007]<...
In conclusion introduction:<br />What are our millennium challenges? <br />What is our relationship with industry 	and wal...
Upcoming SlideShare
Loading in...5
×

The Trouble with Social Computing Systems Research

1,104

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,104
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • The rise of social computing is impacting SIGCHI research immensely. Wikipedia, Twitter, Delicious, and Mechanical Turk have helped us begin to understand people and their interactions through large, naturally occurring datasets. Computational social science will only grow in the coming years.
  • Likewise, those invested in the systems research community in social computing hope for a trajectory of novel, impactful sociotechnical designs. By systems research, we mean research whose main contribution is the presentation of a new sociotechnical artifact, algorithm, design, or platform. Systems research in social computing is valuable because it envisions new ways of interacting with social systems, and spreads these ideas to other researchers and the world at large. This broader focus, married with a massive growth in platforms, APIs, and interest in social computing, might suggest that we will see lots of new interesting research systems.
  • This 4:1 ratio may reflect overall submission ratios to CHI, represent a steady state and not a trend, or equalize out in the long term in terms of highly-cited papers. However, a 4:1 ratio is still worrying: a perceived publication bias might push new researchers capable of both types of work to study systems rather than build them. If this happens en masse, it might threaten our field’s ability to look forward.
  • Note these are not my papers
  • I’m going to argue that ____.
  • Memorable technical contributions are simple ideas that enable interesting, complex scenarios. Systems demos will thus target flashy tasks, aim years ahead of the technology adoption curve, or assume technically literate (often expert) users. For example, end user programming, novel interaction techniques, and augmented reality research all make assumptions about Moore’s Law, adoption, or user training.
  • Memorable technical contributions are simple ideas that enable interesting, complex scenarios. Systems demos will thus target flashy tasks, aim years ahead of the technology adoption curve, or assume technically literate (often expert) users. For example, end user programming, novel interaction techniques, and augmented reality research all make assumptions about Moore’s Law, adoption, or user training.
  • Memorable technical contributions are simple ideas that enable interesting, complex scenarios. Systems demos will thus target flashy tasks, aim years ahead of the technology adoption curve, or assume technically literate (often expert) users. For example, end user programming, novel interaction techniques, and augmented reality research all make assumptions about Moore’s Law, adoption, or user training.
  • Memorable technical contributions are simple ideas that enable interesting, complex scenarios. Systems demos will thus target flashy tasks, aim years ahead of the technology adoption curve, or assume technically literate (often expert) users. For example, end user programming, novel interaction techniques, and augmented reality research all make assumptions about Moore’s Law, adoption, or user training.
  • Memorable technical contributions are simple ideas that enable interesting, complex scenarios. Systems demos will thus target flashy tasks, aim years ahead of the technology adoption curve, or assume technically literate (often expert) users. For example, end user programming, novel interaction techniques, and augmented reality research all make assumptions about Moore’s Law, adoption, or user training.
  • Memorable technical contributions are simple ideas that enable interesting, complex scenarios. Systems demos will thus target flashy tasks, aim years ahead of the technology adoption curve, or assume technically literate (often expert) users. For example, end user programming, novel interaction techniques, and augmented reality research all make assumptions about Moore’s Law, adoption, or user training.
  • The Trouble with Social Computing Systems Research

    1. 1. Michael S. Bernstein, Mark S. Ackerman, Ed H. Chi, Robert C. Miller<br />
    2. 2. Michael S. Bernstein, Mark S. Ackerman, Ed H. Chi, Robert C. Miller<br />
    3. 3. Michael S. Bernstein, Mark S. Ackerman, Ed H. Chi, Robert C. Miller<br />
    4. 4. Ratio of “Understanding Users” papers<br />to “Systems, Tools, Architectures and Infrastructure” papers <br />submitted to the Interaction Beyond the Individual track at CHI 2011.<br />4:1<br />
    5. 5. Trouble: Exponential Growth<br />Your usage data is not really compelling because only a small fraction of Facebook is using the application. Worse, your numbers aren’t growing in anything like an exponential fashion.<br />– CHI metareviewer, paraphrased<br />
    6. 6. Suggestion: Exponential Growth<br />Separate evaluation of spread from steady-state.<br />Which claim is the paper making?<br />
    7. 7. Trouble: Snowball Sampling<br />The authors’ choice of study method – snowball sampling their system by advertising within their own social network – potentially leads to serious problems with validity.<br />– CHI metareviewer, paraphrased<br />
    8. 8. Suggestion: Snowball Sampling<br />Snowballing is inevitablein social systems. It is fundamental to how they operate.<br />
    9. 9.
    10. 10. NoveltyBetween a Rock and a Hard Science<br />
    11. 11. sociotechnical<br />
    12. 12. sociotechnical<br />studiersbuilders<br />
    13. 13. sociotechnical<br />studiersbuilders<br />Fatal Flaw Fallacy [Olsen]<br />Ecological validity at the cost of internal validity<br />[Ackerman 2000], [Barkhuus and Rode 2007], [Chi 2009], [Greenberg and Buxton 2008], [Kaye and Sengers 2007], [Landay 2009], [Lieberman 2003], [Olsen 2007], [Zhai 2003]<br />
    14. 14. sociotechnical<br />studiersbuilders<br />Show us elegant complexity.(simple ideas that enable complex scenarios)<br />We let people type messages up to 140 characters.<br />That’s it? What is possible now that wasn’t before?<br />Nothing — but focus on emergent social activity.<br />Can you add multitouch?<br />Not using IE8.<br />
    15. 15. socio technical<br />studiers builders<br />Build a technically interesting system <br /> (that is hard to spread or evaluate), or<br />Simplify to a system with socially interesting outcomes <br /> (that builders find less novel).<br />
    16. 16. The contribution needs to take one strong stance or another. Either it describes a novel system or a novel social interaction. If it’s a system, then I question the novelty. If it’s a social interaction, it needs more development.<br />– CHI metareviewer, paraphrased<br />Build a technically interesting system <br /> (that is hard to spread or evaluate), or<br />Simplify to a system with socially interesting outcomes <br /> (that builders find less novel).<br />
    17. 17. Create a shared understanding<br />of research contributions<br />
    18. 18. social technical<br />New forms of social interaction<br /> Shared organizational memory [Ackerman 1994]<br />Designs that impact social interactions<br />Increasing online contribution [Beenen et al. 2004]<br />Enable fluent social interaction in a new domain<br />Socially translucent systems [Erickson and Kellogg 2000]<br />
    19. 19. social technical<br />Designs collecting or powered by social data<br />Wikidashboard [Suh et al. 2008]; sense.us [Heeret al. 2007]<br />Algorithms to coordinate crowds or derive signal from social data<br />Collaborative Filtering [Resnick et al. 1994]; Iterate-and-Vote [Little et al. 2010]<br />Platforms and infrastructures<br />TurKit [Little et al. 2010]<br />
    20. 20. social technical<br />×<br />Paired contributions can increase each others’ value<br />ManyEyes<br />[Viégas et al. 2007]<br />
    21. 21. In conclusion introduction:<br />What are our millennium challenges? <br />What is our relationship with industry and walled gardens?<br />How can (and should) we evolve<br />our standards of proof?<br />
    22. 22. Michael S. Bernstein, Mark S. Ackerman, Ed H. Chi, Robert C. Miller<br />
    23. 23.
    24. 24. Make clear that “not in this talk” doesn’t mean “not in the paper”<br />AC quote about it’s not quite this, not quite that, it’s rejected<br />Frame it as Problems with reviewing?<br />Reasons that papers are getting rejected<br />How do we judge these papers? Not “how can we do social computing research”<br />Label as problem or solution<br />Authors’ names<br />Frame contributions of the paper?<br />Three-way conversation<br />Juho: improve the conversation part<br />
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×