• Save
Non-Experimental Methods
Upcoming SlideShare
Loading in...5
×
 

Non-Experimental Methods

on

  • 7,160 views

 

Statistics

Views

Total Views
7,160
Views on SlideShare
7,152
Embed Views
8

Actions

Likes
0
Downloads
0
Comments
0

1 Embed 8

http://www.slideshare.net 8

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Non-Experimental Methods Non-Experimental Methods Presentation Transcript

  • Non-Experimental Methods Kurt Luther
  • Non-Experimental Methods Kurt Luther Experiment-Free
  • Non-Experimental vs. Experiment-Free
    • Childless vs. Childfree
    • Not worse, just different!
  • Why Experiment-Free?
    • Some variables can’t be manipulated…
      • Ethically
      • Practically
      • Period!
    • Experiments often based on experiment-free research
      • Exploratory studies
      • Help identify and scope research questions
  • Characteristics of Experiment-Free Research
  • Experiment-Free Research Methods
    • Naturalistic Observation
    • Ethnography
    • Sociometry
    • The Case History
    • Archival Research
    • Content Analysis
  • Experiment-Free Research Methods
    • Naturalistic Observation
    • Ethnography
    • Sociometry
    • The Case History
    • Archival Research
    • Content Analysis
  • Naturalistic Observation vs. Ethnography
    • Naturalistic Observation
      • Unobtrusive observations of subjects’ naturally occurring behavior are made
    • Ethnography
      • The researcher becomes immersed in the behavioral or social system being studied. May be conducted as a participant or non-participant observation study
  • How to Observe
    • What to record
      • Space, time, materials, names
      • Keep research questions in mind
      • Don’t constrain too much at beginning
    • When to record
      • In-vivo vs. afterwards
    • Recording equipment
      • Notepad
      • Audio recorder
      • Digital camera, video camera
    • Types of notes
      • Jottings
        • Observable events, conversations, etc.
        • Analyses of what you’ve seen
      • Diary
        • Emotional responses
  • Jottings vs. Diary From Jules Henry, Culture Against Man (1963)
    • "Boris had trouble reducing 12 over 16 to the lowest terms and could only get as far as 6 over 8. The teacher asked him quietly if that was as far as he could reduce it. She suggested he 'think'. Much heaving up and down and waving of hands by the other children, all frantic to correct him. Boris pretty unhappy, probably mentally paralyzed. The teacher, quiet, patient ignores the others and with look and voice concentrates on Boris. She says, ' Is there a bigger number than two you can divide into the two parts of the fraction?' After a minute or two, she becomes more urgent, but there is no response from Boris. She then turns to the class and says, ' Well, who can tell Boris what the number is?' A forest of hands appears, and the teacher calls Peggy. Peggy says that four may be divided into the numerator and the denominator.”
    • “ Boris' failure has made it possible for Peggy to succeed; his depression is the price of her exhilaration. This is the standard condition of American elementary schools…so often somebody's success is bought at the cost of our failure. To a Zuni, Hopi or Dakota Indian Peggy's performance would be viewed as cruel beyond belief, for competition, the writing of success from somebody’s failure, is a form of torture foreign to those non-competitive [cultures]. Yet Peggy’s action seems natural to us; and so it is. How else would you run the world? And since all but the brightest have the constant experience that others succeed at their expense they cannot but develop an inherent tendency to hate -- to hate the success of others, to hate the successful and become determined to prevent it. Along with this, naturally, goes the hope that others will fail…”
  • History of Ethnography
    • 19 th Century
      • Origin in anthropology
      • Western Expansionism
        • North America: Boas and Native Americans
        • Europe: Colonial encounters
      • Mostly “armchair anthropology”
    • 20 th and 21 st Century
      • Malinowski and Trobriand Islands
      • Sociology and Chicago School
        • Looking inward
        • Marginalized populations
          • Prostitutes, prisoners, homeless, mentally ill
      • Modern ethnography
        • Subcultures
        • Technological adoption
          • Within organizations
          • Virtual ethnography (online communities)
  •  
  • “ Pass-my-Flash 2” An example collaborative animation project (collab) http://www.newgrounds.com/portal/view/428554
  • Some Numbers
    • Year created: 2008
    • Audience views: 65,000+
    • Participants: 7
      • Ages: 17 to 29
      • Countries: 2
        • U.S. and CAN
      • Cities: 5
        • Brooklyn, Ontario, Orlando, Savannah, Washington D.C.
  • Some Questions
    • How did these animators meet ?
    • How did they agree on a collab?
    • How was the work divided up ?
    • How much planning was involved?
    • What roles , if any, did animators adopt?
    • What other types of collabs exist?
    • How does all of this compare with other types of online, collaboratively-created projects?
  • Study
    • Sites
      • Three online animation communities
        • Primarily Newgrounds.com
          • Created in 1995 by Tom Fulp
          • 1,500,000+ registered members
          • 130,000+ member-contributed animations
        • Two other “satellite communities”
          • < 500 members
          • Contribute to Newgrounds.com
    • Scope
      • Focused on movie collabs
  • Research Questions
    • How do people collaborate over the Internet to create animated short movies? ( i.e. , what is the “collab production process”?)
    • What challenges must leaders manage throughout the collab production process and how do leaders manage these challenges?
    • How might technology be designed to help collab leaders manage these challenges?
  •  
  •  
  • Interviewing
    • Technically, in-depth, phenomenological interviewing
    • Why?
      • Understand experiences of collab leaders
      • Descriptive account of collab production process
    • Recruitment
      • Purposeful sampling (collab participants)
      • Via discussion forums, private messages, “snowball sampling”
    • Participants
      • 14 telephone, 3 email
      • Ages: 16 to 29; all male
      • 6 countries (USA, UK, Australia, Spain, Netherlands, Estonia)
      • Real names vs. pseudonyms [Bruckman 2002]
    • Questions
      • Start with generic interview guide
      • Semi-structured format
  • Participant Demographics * Led at least one collab † Pseudonym Name Age Country Joseph Blanchette * 24 U.S. Eric Carlson 19 U.S. Luis Castanon * 27 U.S. Michael Frank * 19 U.S. Tom Fulp * 29 U.S. James Hole * 16 Australia Tyler Koch * 19 U.S. Massimo Maitan * 21 Australia Anders-Martin Meister 16 Estonia Ross O’Donovan * 19 Australia Kraig Phillips 27 U.S. Joseph Rooks * 21 U.S. Kester Smith 21 U.K. Anonymous 1 † 18 Netherlands Hans Van Harken * 17 Spain Robert Westgate 21 U.K. Anonymous 2 † 19 U.S.
  •  
  • > > Hello there, my name is Dan, I'm 14, and I just found a thread on a > > forum I go to on a regular basis, and the person who posted it > > claims to be a part of GVU, and it doing interviews on &quot;Creative > > Collaborations in Online Communities&quot;, and he wants to do all the > > interviews over the phone. Naturally, I am a little skeptical on > > whether or not this is a scam or not. I checked through the names > > of the students currently in GVU, but he wasn't there. He claims to > > be a Mr. Kurt Luther, and if you want, here's the link to the thread: > > http://ngcollabs.galacti.com/bbs/viewtopic.php?t=2586 > > > > there's a few reasons why I'm a bit worried about it: > > 1) As I said earlier, I couldn't find him in the enlisted students. > > 2) He won't do interviews over Instant Messanging sevices, and > > insists they be over the phone > > 3) almost none of the people at NG Collabs are over 18 > > 4) the field he is reasearching is so limited > > 5) we are a very small community, with only about 50 active members > > 6) www.newgrounds.com/bbs has a much bigger forum for flash > > collaborations, and would get much more response, not to mention > > one of the only ways to get to NG Collabs is through Newgrounds, > > seeing as we are trying to have our site hosted by them > > > > So I am very worried about this, and so I wanted some proof that > > Kurt Luther actually does exist. > > If you could get back to me sometime on this matter, I would > > appreciate it. > > > > Thank you for your time, > > Dan
  •  
  •  
  •  
  • Grounded Theory Analysis
    • Structured approach to analyzing qualitative data
      • Often used for analysis of interview data
      • Allows themes to emerge from the data
    • Bottom-up coding
      • Open coding
        • Label phenomena (concepts)
        • Discover categories (groups of concepts)
      • Axial coding
        • Make connections between categories
      • Selective coding
        • Select the core category (story line)
        • Relate other categories to the story line
    • Result
      • Series of interrelated themes telling one coherent story
  • Questions?
    • Acknowledgments
      • Interviewees, Newgrounds.com community, and Tom Fulp
      • Anonymous CSCW 2008 reviewers
      • ELC Lab, especially Sarita Yardi
      • Kelly Caine, Pam Griffith, Beki Grinter, and Shruthi Panicker, Kevin Ziegler
      • National Science Foundation
      • You (thanks for listening!)
    • Contact us
      • Kurt Luther
        • [email_address]
      • Amy Bruckman
        • [email_address]
  • Ethnography Issues
    • Observing as a participant or non-participant
    • Gaining access to a field setting
    • Gaining entry into the group
    • Becoming invisible vs. “going native”
    • Making observations and recording data
    • Analyzing ethnographic data
  • Experiment-Free Research Methods
    • Naturalistic Observation
    • Ethnography
    • Sociometry
    • The Case History
    • Archival Research
    • Content Analysis
  • Content Analysis Basics
    • Used to analyze a written or spoken record for occurrence of specific behaviors or events
    • Archival sources often used as sources for data
    • Appears simple, but may be complex
    • Should be used within a clearly developed study, including hypotheses to be tested
    • Response categories must be clearly defined
    • A method for quantifying behavior must be defined
  • Performing a Content Analysis
    • Clearly defined response categories are essential
    • Two units of analysis
      • Recording unit: Element of the material you are going to record (e.g., instances of a certain word)
      • Context unit: Context within which material analyzed appears
    • Observers doing content analysis must be blind so that bias will not enter the analysis
    • Materials to be analyzed should be chosen carefully to increase generality
    • Cannot be used to establish causal connections among variables
  • Content Analysis of Collab Threads
    • Why?
      • How many collabs result in a completed animation? How many do not? Why?
    • Content
      • All collab activity is kept public
      • Collab threads
        • “ Screen scraping” via Python script/MySQL database
        • ~ 1,600 between Sept. 2003 to Sept. 2006
    • Criteria
      • Complete or incomplete? Why?
    • Coding
      • Performed manually by two judges
      • Calculated inter-rater reliability
  •  
  •  
  •  
  • Advanced Content Analysis
    • Detailed content analysis of collab threads
      • “ Screen scraped” all Newgrounds.com collab threads
        • 150,000+ posts total
        • Manually coding a random sample
    • Why?
      • Determine completion rates for different types of collabs
      • Attributes of completed vs. incomplete collabs
        • What does a “likely to succeed” collab look like?
          • Leadership style, specs, arrangements, themes, etc.
      • Understand how collab structures change over time
  •  
  • Findings
    • Few collabs (<20%) are completed
    • 3 major challenges for leaders
      • Designing the project
        • Structuring, proposing
      • Managing the artists
        • Recruiting, directing, motivating, replacing
      • Completing the project
        • Integrating, publishing
  • Findings
    • Few collabs (<20%) are completed
    • 3 major challenges for leaders
      • Designing the project
        • Structuring , proposing
      • Managing the artists
        • Recruiting, directing , motivating, replacing
      • Completing the project
        • Integrating , publishing
  • Structuring the Project
    • Modularization and granularity [Parnas 1972; Benkler 2006]
    “ Every author on Newgrounds likes to make their own thing. You wouldn’t be able to give one person a job of storyboarding and one person a job of animating it and one person a job of recording the sound. It just wouldn’t work. People have to do their own thing on Newgrounds, so it’s a lot easier to just let them do their piece of animation and take a whole month to do it.” (Massimo Maitan)
  • Structuring the Project
    • Specs
      • Technical guidelines
      • e.g. , frame rate, dimensions
    • Themes
      • Content guidelines
      • e.g. , narrative, music, visual element, event, vignette
    • Arrangements
      • Collaboration guidelines
      • Linear, continuous, nonlinear
    “ [Y]ou have to give people really concrete boundaries in terms of how to put their movie together—not the creative part, but the technical aspects of it—in order for it to succeed to begin with.” (Luis Castanon) “ If people are more free, people will come up with a huge range of ideas, and people won’t get bored over the course of the Flash. But at the same time, they’ve got to be restrictive enough so that when the Flash is put together, it works.” (Robert Westgate)
  •  
  •  
  • Structuring the Project
    • Specs
      • Technical guidelines
      • e.g. , frame rate, dimensions
    • Themes
      • Content guidelines
      • e.g. , narrative, music, visual element, event, vignette
    • Arrangements
      • Collaboration guidelines
      • Linear, continuous, nonlinear
    “ [Y]ou have to give people really concrete boundaries in terms of how to put their movie together—not the creative part, but the technical aspects of it—in order for it to succeed to begin with.” (Luis Castanon) “ If people are more free, people will come up with a huge range of ideas, and people won’t get bored over the course of the Flash. But at the same time, they’ve got to be restrictive enough so that when the Flash is put together, it works.” (Robert Westgate)
  • “ Valentine ‘29” Linear Story Collab
  • “ Pass-my-Flash 2” Continuous Story Collab
  • “ When Farm Animals Attack” Nonlinear Story Collab
  • More Examples Concept Example Arrangement Theme Theme Title Yr. # Parts. # Auths. Participant(s) Linear Narrative chapter Events of St. Valentine’s Day Massacre “ Valentine ‘29”† 2007 6 6 Hans Van Harken* Music lyrics “ One O’Clock Jump” by Tommy and Jimmy Dorsey “ One O’Clock Jump”† 2005 11 9 Luis Castanon* “ Stop Crying Your Heart Out” by Oasis “ Stop Cryin Your Heart Out”† 2005 24 4 Luis Castanon “ We Hate When Our Friends Become Successful” by Morrissey “ We Hate It When…”† 2005 16 5 Ross O’Donovan Genre selection Television series episode “ The Clockcrew TV Collab 2”† 2007 50+ 1 Joseph Rooks* Improvisational Continuous visual element Red line “ The Red Line”† 2007 50+ 1 Robert Westgate Mountain slope “ Mount Newgrounds” 2007 90+ 2 Robert Westgate Nonlinear Vignette prompt Farm animals attacking people “ The ‘W.F.A.A.’ Collab” 2006 10 5 Massimo Maitan* Pirates or ninjas “ NG TimeTrial IX”† 2004 4 4 Luis Castanon* Emoticons “‘ Emoticonisation’ Collab”† 2006 12 5 Massimo Maitan* Make fun of Mr_Artist “ The Mr. Artist Collab” 2005 5 4 Michael Frank* Cooking “ Cooking with Clockcrew”† 2006 7 7 Tyler Koch* Pop culture parody Transformers film parodies “ Blamformers!”† 2007 30+ 9 Tom Fulp* The Matrix film parodies “ The Matrix Still Has You”† 2004 5 5 Joe Blanchette* Halo 2 videogame parodies “ TSAH”† 2004 6 5 Ross O’Donovan* Halo 2 videogame parodies “ TSAH2”† 2006 5 5 Joe Blanchette Format/style constraint Pixilated graphics “ Retro Collab”† 2007 14 10 James Hole* Frame-by-frame animation style “ NG TT Series Finale”† 2005 7 7 Luis Castanon* Time constraint Complete animation within 3 days “ NG Time Trial Challenge”† 2004 10 4 Luis Castanon*
  • Directing the Artists
    • Everyone’s a volunteer
    • Authorial leadership
      • Balance authority and egalitarianism
      • Too much
        • Artists drop out
      • Too little
        • Under-utilization
        • Collab never completed
    • Commitment
      • Leaders can’t quit
        • Creative vision doesn’t transfer
        • Logistical issues
    “ If you’re collaborating, you gotta make everybody feel like they’re a part of it. You’ve got to make them feel like it’s all their movie. Because if it’s not, then they won’t want to work on it.” (Tyler Koch) “ There have been times where people have said, ‘Yeah, you’re too bossy.’ And I’m like, ‘Yeah, but if I’m not bossy, you guys are never gonna get it done.’” (Ross O’Donovan) “ I don’t think of it as a position of power. I think of it as a position that enables me to…give them things to participate in.” (Joseph Rooks)
  • Integrating the Animations
    • Recomposition
    • Integration challenges
      • Aesthetic
        • Variety vs. continuity
      • Social
        • Ownership
      • Technical
        • Compilation issues
          • File size
          • Flash symbols
    “ [Variety is] one of the best parts about collaborations, different people’s art styles coming together…but it’s still important [that], like, in some way it flows.” (James Hole) “ How would you feel if someone…changed your work without telling you? It’s just…you should at least inform the person.” (Anders-Martin Meister) “ The hard thing for me, working on [collabs], is dealing with other people’s techniques and methods for making stuff.” (Tom Fulp)
  •  
  • Contributions
    • An empirically-grounded description of practices surrounding online creative collaboration in the open-ended problem domain of entertainment
    • A characterization of three major challenges faced by online creative collaboration leaders while creating entertainment and how they manage these challenges.
    • A discussion of these challenges vis-à-vis those faced by leaders of other forms of online creative collaboration
  • What’s Next
    • New tool: Sandbox
      • Web-based collaborative system for supporting online creative collaboration
      • Focus on the context of animation production
    • Two modes
      • Sandbox Planner
      • Sandbox Improv
    • Evaluation
      • Compare and contrast both Sandbox modes with existing leadership practices
      • Mixed methods
        • Log files, content analysis, and in-depth interviews
  • Learn More
    • Research Paper: “Leadership in Online Creative Collaboration”
      • Authors: Kurt Luther and Amy Bruckman
      • URL: http://doi.acm.org/10.1145/1460563.1460619
    • Course: CS 4690 Empirical Methods for HCI
      • Instructor: Professor Rebecca Grinter
      • URL: http://www.cc.gatech.edu/~beki/cs4690/CS4690_CS6455.html
      • Lecture notes are online
    • My email: [email_address]