Non-Experimental Methods Kurt Luther

Non-Experimental Methods Kurt Luther Experiment-Free

Non-Experimental vs. Experiment-Free Childless vs. Childfree Not worse, just different!

Childless vs. Childfree

Not worse, just different!

Why Experiment-Free? Some variables can’t be manipulated… Ethically Practically Period! Experiments often based on experiment-free research Exploratory studies Help identify and scope research questions

Some variables can’t be manipulated…

Ethically

Practically

Period!

Experiments often based on experiment-free research

Exploratory studies

Help identify and scope research questions

Characteristics of Experiment-Free Research

Experiment-Free Research Methods Naturalistic Observation Ethnography Sociometry The Case History Archival Research Content Analysis

Naturalistic Observation

Ethnography

Sociometry

The Case History

Archival Research

Content Analysis

Experiment-Free Research Methods Naturalistic Observation Ethnography Sociometry The Case History Archival Research Content Analysis

Naturalistic Observation

Ethnography

Sociometry

The Case History

Archival Research

Content Analysis

Naturalistic Observation vs. Ethnography Naturalistic Observation Unobtrusive observations of subjects’ naturally occurring behavior are made Ethnography The researcher becomes immersed in the behavioral or social system being studied. May be conducted as a participant or non-participant observation study

Naturalistic Observation

Unobtrusive observations of subjects’ naturally occurring behavior are made

Ethnography

The researcher becomes immersed in the behavioral or social system being studied. May be conducted as a participant or non-participant observation study

How to Observe What to record Space, time, materials, names Keep research questions in mind Don’t constrain too much at beginning When to record In-vivo vs. afterwards Recording equipment Notepad Audio recorder Digital camera, video camera Types of notes Jottings Observable events, conversations, etc. Analyses of what you’ve seen Diary Emotional responses

What to record

Space, time, materials, names

Keep research questions in mind

Don’t constrain too much at beginning

When to record

In-vivo vs. afterwards

Recording equipment

Notepad

Audio recorder

Digital camera, video camera

Types of notes

Jottings

Observable events, conversations, etc.

Analyses of what you’ve seen

Diary

Emotional responses

Jottings vs. Diary From Jules Henry, Culture Against Man (1963) "Boris had trouble reducing 12 over 16 to the lowest terms and could only get as far as 6 over 8. The teacher asked him quietly if that was as far as he could reduce it. She suggested he 'think'. Much heaving up and down and waving of hands by the other children, all frantic to correct him. Boris pretty unhappy, probably mentally paralyzed. The teacher, quiet, patient ignores the others and with look and voice concentrates on Boris. She says, ' Is there a bigger number than two you can divide into the two parts of the fraction?' After a minute or two, she becomes more urgent, but there is no response from Boris. She then turns to the class and says, ' Well, who can tell Boris what the number is?' A forest of hands appears, and the teacher calls Peggy. Peggy says that four may be divided into the numerator and the denominator.” “ Boris' failure has made it possible for Peggy to succeed; his depression is the price of her exhilaration. This is the standard condition of American elementary schools…so often somebody's success is bought at the cost of our failure. To a Zuni, Hopi or Dakota Indian Peggy's performance would be viewed as cruel beyond belief, for competition, the writing of success from somebody’s failure, is a form of torture foreign to those non-competitive [cultures]. Yet Peggy’s action seems natural to us; and so it is. How else would you run the world? And since all but the brightest have the constant experience that others succeed at their expense they cannot but develop an inherent tendency to hate -- to hate the success of others, to hate the successful and become determined to prevent it. Along with this, naturally, goes the hope that others will fail…”

"Boris had trouble reducing 12 over 16 to the lowest terms and could only get as far as 6 over 8. The teacher asked him quietly if that was as far as he could reduce it. She suggested he 'think'. Much heaving up and down and waving of hands by the other children, all frantic to correct him. Boris pretty unhappy, probably mentally paralyzed. The teacher, quiet, patient ignores the others and with look and voice concentrates on Boris. She says, ' Is there a bigger number than two you can divide into the two parts of the fraction?' After a minute or two, she becomes more urgent, but there is no response from Boris. She then turns to the class and says, ' Well, who can tell Boris what the number is?' A forest of hands appears, and the teacher calls Peggy. Peggy says that four may be divided into the numerator and the denominator.”

“ Boris' failure has made it possible for Peggy to succeed; his depression is the price of her exhilaration. This is the standard condition of American elementary schools…so often somebody's success is bought at the cost of our failure. To a Zuni, Hopi or Dakota Indian Peggy's performance would be viewed as cruel beyond belief, for competition, the writing of success from somebody’s failure, is a form of torture foreign to those non-competitive [cultures]. Yet Peggy’s action seems natural to us; and so it is. How else would you run the world? And since all but the brightest have the constant experience that others succeed at their expense they cannot but develop an inherent tendency to hate -- to hate the success of others, to hate the successful and become determined to prevent it. Along with this, naturally, goes the hope that others will fail…”

History of Ethnography 19 th Century Origin in anthropology Western Expansionism North America: Boas and Native Americans Europe: Colonial encounters Mostly “armchair anthropology” 20 th and 21 st Century Malinowski and Trobriand Islands Sociology and Chicago School Looking inward Marginalized populations Prostitutes, prisoners, homeless, mentally ill Modern ethnography Subcultures Technological adoption Within organizations Virtual ethnography (online communities)

19 th Century

Origin in anthropology

Western Expansionism

North America: Boas and Native Americans

Europe: Colonial encounters

Mostly “armchair anthropology”

20 th and 21 st Century

Malinowski and Trobriand Islands

Sociology and Chicago School

Looking inward

Marginalized populations

Prostitutes, prisoners, homeless, mentally ill

Modern ethnography

Subcultures

Technological adoption

Within organizations

Virtual ethnography (online communities)

“ Pass-my-Flash 2” An example collaborative animation project (collab) http://www.newgrounds.com/portal/view/428554

Some Numbers Year created: 2008 Audience views: 65,000+ Participants: 7 Ages: 17 to 29 Countries: 2 U.S. and CAN Cities: 5 Brooklyn, Ontario, Orlando, Savannah, Washington D.C.

Year created: 2008

Audience views: 65,000+

Participants: 7

Ages: 17 to 29

Countries: 2

U.S. and CAN

Cities: 5

Brooklyn, Ontario, Orlando, Savannah, Washington D.C.

Some Questions How did these animators meet ? How did they agree on a collab? How was the work divided up ? How much planning was involved? What roles , if any, did animators adopt? What other types of collabs exist? How does all of this compare with other types of online, collaboratively-created projects?

How did these animators meet ?

How did they agree on a collab?

How was the work divided up ?

How much planning was involved?

What roles , if any, did animators adopt?

What other types of collabs exist?

How does all of this compare with other types of online, collaboratively-created projects?

Study Sites Three online animation communities Primarily Newgrounds.com Created in 1995 by Tom Fulp 1,500,000+ registered members 130,000+ member-contributed animations Two other “satellite communities” < 500 members Contribute to Newgrounds.com Scope Focused on movie collabs

Sites

Three online animation communities

Primarily Newgrounds.com

Created in 1995 by Tom Fulp

1,500,000+ registered members

130,000+ member-contributed animations

Two other “satellite communities”

< 500 members

Contribute to Newgrounds.com

Scope

Focused on movie collabs

Research Questions How do people collaborate over the Internet to create animated short movies? ( i.e. , what is the “collab production process”?) What challenges must leaders manage throughout the collab production process and how do leaders manage these challenges? How might technology be designed to help collab leaders manage these challenges?

How do people collaborate over the Internet to create animated short movies? ( i.e. , what is the “collab production process”?)

What challenges must leaders manage throughout the collab production process and how do leaders manage these challenges?

How might technology be designed to help collab leaders manage these challenges?

Interviewing Technically, in-depth, phenomenological interviewing Why? Understand experiences of collab leaders Descriptive account of collab production process Recruitment Purposeful sampling (collab participants) Via discussion forums, private messages, “snowball sampling” Participants 14 telephone, 3 email Ages: 16 to 29; all male 6 countries (USA, UK, Australia, Spain, Netherlands, Estonia) Real names vs. pseudonyms [Bruckman 2002] Questions Start with generic interview guide Semi-structured format

Technically, in-depth, phenomenological interviewing

Why?

Understand experiences of collab leaders

Descriptive account of collab production process

Recruitment

Purposeful sampling (collab participants)

Via discussion forums, private messages, “snowball sampling”

Participants

14 telephone, 3 email

Ages: 16 to 29; all male

6 countries (USA, UK, Australia, Spain, Netherlands, Estonia)

Real names vs. pseudonyms [Bruckman 2002]

Questions

Start with generic interview guide

Semi-structured format

Participant Demographics * Led at least one collab † Pseudonym Name Age Country Joseph Blanchette * 24 U.S. Eric Carlson 19 U.S. Luis Castanon * 27 U.S. Michael Frank * 19 U.S. Tom Fulp * 29 U.S. James Hole * 16 Australia Tyler Koch * 19 U.S. Massimo Maitan * 21 Australia Anders-Martin Meister 16 Estonia Ross O’Donovan * 19 Australia Kraig Phillips 27 U.S. Joseph Rooks * 21 U.S. Kester Smith 21 U.K. Anonymous 1 † 18 Netherlands Hans Van Harken * 17 Spain Robert Westgate 21 U.K. Anonymous 2 † 19 U.S.

> > Hello there, my name is Dan, I'm 14, and I just found a thread on a > > forum I go to on a regular basis, and the person who posted it > > claims to be a part of GVU, and it doing interviews on &quot;Creative > > Collaborations in Online Communities&quot;, and he wants to do all the > > interviews over the phone. Naturally, I am a little skeptical on > > whether or not this is a scam or not. I checked through the names > > of the students currently in GVU, but he wasn't there. He claims to > > be a Mr. Kurt Luther, and if you want, here's the link to the thread: > > http://ngcollabs.galacti.com/bbs/viewtopic.php?t=2586 > > > > there's a few reasons why I'm a bit worried about it: > > 1) As I said earlier, I couldn't find him in the enlisted students. > > 2) He won't do interviews over Instant Messanging sevices, and > > insists they be over the phone > > 3) almost none of the people at NG Collabs are over 18 > > 4) the field he is reasearching is so limited > > 5) we are a very small community, with only about 50 active members > > 6) www.newgrounds.com/bbs has a much bigger forum for flash > > collaborations, and would get much more response, not to mention > > one of the only ways to get to NG Collabs is through Newgrounds, > > seeing as we are trying to have our site hosted by them > > > > So I am very worried about this, and so I wanted some proof that > > Kurt Luther actually does exist. > > If you could get back to me sometime on this matter, I would > > appreciate it. > > > > Thank you for your time, > > Dan

Grounded Theory Analysis Structured approach to analyzing qualitative data Often used for analysis of interview data Allows themes to emerge from the data Bottom-up coding Open coding Label phenomena (concepts) Discover categories (groups of concepts) Axial coding Make connections between categories Selective coding Select the core category (story line) Relate other categories to the story line Result Series of interrelated themes telling one coherent story

Structured approach to analyzing qualitative data

Often used for analysis of interview data

Allows themes to emerge from the data

Bottom-up coding

Open coding

Label phenomena (concepts)

Discover categories (groups of concepts)

Axial coding

Make connections between categories

Selective coding

Select the core category (story line)

Relate other categories to the story line

Result

Series of interrelated themes telling one coherent story

Questions? Acknowledgments Interviewees, Newgrounds.com community, and Tom Fulp Anonymous CSCW 2008 reviewers ELC Lab, especially Sarita Yardi Kelly Caine, Pam Griffith, Beki Grinter, and Shruthi Panicker, Kevin Ziegler National Science Foundation You (thanks for listening!) Contact us Kurt Luther [email_address] Amy Bruckman [email_address]

Acknowledgments

Interviewees, Newgrounds.com community, and Tom Fulp

Anonymous CSCW 2008 reviewers

ELC Lab, especially Sarita Yardi

Kelly Caine, Pam Griffith, Beki Grinter, and Shruthi Panicker, Kevin Ziegler

National Science Foundation

You (thanks for listening!)

Contact us

Kurt Luther

[email_address]

Amy Bruckman

[email_address]

Ethnography Issues Observing as a participant or non-participant Gaining access to a field setting Gaining entry into the group Becoming invisible vs. “going native” Making observations and recording data Analyzing ethnographic data

Observing as a participant or non-participant

Gaining access to a field setting

Gaining entry into the group

Becoming invisible vs. “going native”

Making observations and recording data

Analyzing ethnographic data

Experiment-Free Research Methods Naturalistic Observation Ethnography Sociometry The Case History Archival Research Content Analysis

Naturalistic Observation

Ethnography

Sociometry

The Case History

Archival Research

Content Analysis

Content Analysis Basics Used to analyze a written or spoken record for occurrence of specific behaviors or events Archival sources often used as sources for data Appears simple, but may be complex Should be used within a clearly developed study, including hypotheses to be tested Response categories must be clearly defined A method for quantifying behavior must be defined

Used to analyze a written or spoken record for occurrence of specific behaviors or events

Archival sources often used as sources for data

Appears simple, but may be complex

Should be used within a clearly developed study, including hypotheses to be tested

Response categories must be clearly defined

A method for quantifying behavior must be defined

Performing a Content Analysis Clearly defined response categories are essential Two units of analysis Recording unit: Element of the material you are going to record (e.g., instances of a certain word) Context unit: Context within which material analyzed appears Observers doing content analysis must be blind so that bias will not enter the analysis Materials to be analyzed should be chosen carefully to increase generality Cannot be used to establish causal connections among variables

Clearly defined response categories are essential

Two units of analysis

Recording unit: Element of the material you are going to record (e.g., instances of a certain word)

Context unit: Context within which material analyzed appears

Observers doing content analysis must be blind so that bias will not enter the analysis

Materials to be analyzed should be chosen carefully to increase generality

Cannot be used to establish causal connections among variables

Content Analysis of Collab Threads Why? How many collabs result in a completed animation? How many do not? Why? Content All collab activity is kept public Collab threads “ Screen scraping” via Python script/MySQL database ~ 1,600 between Sept. 2003 to Sept. 2006 Criteria Complete or incomplete? Why? Coding Performed manually by two judges Calculated inter-rater reliability

Why?

How many collabs result in a completed animation? How many do not? Why?

Content

All collab activity is kept public

Collab threads

“ Screen scraping” via Python script/MySQL database

~ 1,600 between Sept. 2003 to Sept. 2006

Criteria

Complete or incomplete? Why?

Coding

Performed manually by two judges

Calculated inter-rater reliability

Advanced Content Analysis Detailed content analysis of collab threads “ Screen scraped” all Newgrounds.com collab threads 150,000+ posts total Manually coding a random sample Why? Determine completion rates for different types of collabs Attributes of completed vs. incomplete collabs What does a “likely to succeed” collab look like? Leadership style, specs, arrangements, themes, etc. Understand how collab structures change over time

Detailed content analysis of collab threads

“ Screen scraped” all Newgrounds.com collab threads

150,000+ posts total

Manually coding a random sample

Why?

Determine completion rates for different types of collabs

Attributes of completed vs. incomplete collabs

What does a “likely to succeed” collab look like?

Leadership style, specs, arrangements, themes, etc.

Understand how collab structures change over time

Findings Few collabs (<20%) are completed 3 major challenges for leaders Designing the project Structuring, proposing Managing the artists Recruiting, directing, motivating, replacing Completing the project Integrating, publishing

Few collabs (<20%) are completed

3 major challenges for leaders

Designing the project

Structuring, proposing

Managing the artists

Recruiting, directing, motivating, replacing

Completing the project

Integrating, publishing

Findings Few collabs (<20%) are completed 3 major challenges for leaders Designing the project Structuring , proposing Managing the artists Recruiting, directing , motivating, replacing Completing the project Integrating , publishing

Few collabs (<20%) are completed

3 major challenges for leaders

Designing the project

Structuring , proposing

Managing the artists

Recruiting, directing , motivating, replacing

Completing the project

Integrating , publishing

Structuring the Project Modularization and granularity [Parnas 1972; Benkler 2006] “ Every author on Newgrounds likes to make their own thing. You wouldn’t be able to give one person a job of storyboarding and one person a job of animating it and one person a job of recording the sound. It just wouldn’t work. People have to do their own thing on Newgrounds, so it’s a lot easier to just let them do their piece of animation and take a whole month to do it.” (Massimo Maitan)

Modularization and granularity [Parnas 1972; Benkler 2006]

Structuring the Project Specs Technical guidelines e.g. , frame rate, dimensions Themes Content guidelines e.g. , narrative, music, visual element, event, vignette Arrangements Collaboration guidelines Linear, continuous, nonlinear “ [Y]ou have to give people really concrete boundaries in terms of how to put their movie together—not the creative part, but the technical aspects of it—in order for it to succeed to begin with.” (Luis Castanon) “ If people are more free, people will come up with a huge range of ideas, and people won’t get bored over the course of the Flash. But at the same time, they’ve got to be restrictive enough so that when the Flash is put together, it works.” (Robert Westgate)

Specs

Technical guidelines

e.g. , frame rate, dimensions

Themes

Content guidelines

e.g. , narrative, music, visual element, event, vignette

Arrangements

Collaboration guidelines

Linear, continuous, nonlinear

Structuring the Project Specs Technical guidelines e.g. , frame rate, dimensions Themes Content guidelines e.g. , narrative, music, visual element, event, vignette Arrangements Collaboration guidelines Linear, continuous, nonlinear “ [Y]ou have to give people really concrete boundaries in terms of how to put their movie together—not the creative part, but the technical aspects of it—in order for it to succeed to begin with.” (Luis Castanon) “ If people are more free, people will come up with a huge range of ideas, and people won’t get bored over the course of the Flash. But at the same time, they’ve got to be restrictive enough so that when the Flash is put together, it works.” (Robert Westgate)

Specs

Technical guidelines

e.g. , frame rate, dimensions

Themes

Content guidelines

e.g. , narrative, music, visual element, event, vignette

Arrangements

Collaboration guidelines

Linear, continuous, nonlinear

“ Valentine ‘29” Linear Story Collab

“ Pass-my-Flash 2” Continuous Story Collab

“ When Farm Animals Attack” Nonlinear Story Collab

More Examples Concept Example Arrangement Theme Theme Title Yr. # Parts. # Auths. Participant(s) Linear Narrative chapter Events of St. Valentine’s Day Massacre “ Valentine ‘29”† 2007 6 6 Hans Van Harken* Music lyrics “ One O’Clock Jump” by Tommy and Jimmy Dorsey “ One O’Clock Jump”† 2005 11 9 Luis Castanon* “ Stop Crying Your Heart Out” by Oasis “ Stop Cryin Your Heart Out”† 2005 24 4 Luis Castanon “ We Hate When Our Friends Become Successful” by Morrissey “ We Hate It When…”† 2005 16 5 Ross O’Donovan Genre selection Television series episode “ The Clockcrew TV Collab 2”† 2007 50+ 1 Joseph Rooks* Improvisational Continuous visual element Red line “ The Red Line”† 2007 50+ 1 Robert Westgate Mountain slope “ Mount Newgrounds” 2007 90+ 2 Robert Westgate Nonlinear Vignette prompt Farm animals attacking people “ The ‘W.F.A.A.’ Collab” 2006 10 5 Massimo Maitan* Pirates or ninjas “ NG TimeTrial IX”† 2004 4 4 Luis Castanon* Emoticons “‘ Emoticonisation’ Collab”† 2006 12 5 Massimo Maitan* Make fun of Mr_Artist “ The Mr. Artist Collab” 2005 5 4 Michael Frank* Cooking “ Cooking with Clockcrew”† 2006 7 7 Tyler Koch* Pop culture parody Transformers film parodies “ Blamformers!”† 2007 30+ 9 Tom Fulp* The Matrix film parodies “ The Matrix Still Has You”† 2004 5 5 Joe Blanchette* Halo 2 videogame parodies “ TSAH”† 2004 6 5 Ross O’Donovan* Halo 2 videogame parodies “ TSAH2”† 2006 5 5 Joe Blanchette Format/style constraint Pixilated graphics “ Retro Collab”† 2007 14 10 James Hole* Frame-by-frame animation style “ NG TT Series Finale”† 2005 7 7 Luis Castanon* Time constraint Complete animation within 3 days “ NG Time Trial Challenge”† 2004 10 4 Luis Castanon*

Directing the Artists Everyone’s a volunteer Authorial leadership Balance authority and egalitarianism Too much Artists drop out Too little Under-utilization Collab never completed Commitment Leaders can’t quit Creative vision doesn’t transfer Logistical issues “ If you’re collaborating, you gotta make everybody feel like they’re a part of it. You’ve got to make them feel like it’s all their movie. Because if it’s not, then they won’t want to work on it.” (Tyler Koch) “ There have been times where people have said, ‘Yeah, you’re too bossy.’ And I’m like, ‘Yeah, but if I’m not bossy, you guys are never gonna get it done.’” (Ross O’Donovan) “ I don’t think of it as a position of power. I think of it as a position that enables me to…give them things to participate in.” (Joseph Rooks)

Everyone’s a volunteer

Authorial leadership

Balance authority and egalitarianism

Too much

Artists drop out

Too little

Under-utilization

Collab never completed

Commitment

Leaders can’t quit

Creative vision doesn’t transfer

Logistical issues

Integrating the Animations Recomposition Integration challenges Aesthetic Variety vs. continuity Social Ownership Technical Compilation issues File size Flash symbols “ [Variety is] one of the best parts about collaborations, different people’s art styles coming together…but it’s still important [that], like, in some way it flows.” (James Hole) “ How would you feel if someone…changed your work without telling you? It’s just…you should at least inform the person.” (Anders-Martin Meister) “ The hard thing for me, working on [collabs], is dealing with other people’s techniques and methods for making stuff.” (Tom Fulp)

Recomposition

Integration challenges

Aesthetic

Variety vs. continuity

Social

Ownership

Technical

Compilation issues

File size

Flash symbols

Contributions An empirically-grounded description of practices surrounding online creative collaboration in the open-ended problem domain of entertainment A characterization of three major challenges faced by online creative collaboration leaders while creating entertainment and how they manage these challenges. A discussion of these challenges vis-à-vis those faced by leaders of other forms of online creative collaboration

An empirically-grounded description of practices surrounding online creative collaboration in the open-ended problem domain of entertainment

A characterization of three major challenges faced by online creative collaboration leaders while creating entertainment and how they manage these challenges.

A discussion of these challenges vis-à-vis those faced by leaders of other forms of online creative collaboration

What’s Next New tool: Sandbox Web-based collaborative system for supporting online creative collaboration Focus on the context of animation production Two modes Sandbox Planner Sandbox Improv Evaluation Compare and contrast both Sandbox modes with existing leadership practices Mixed methods Log files, content analysis, and in-depth interviews

New tool: Sandbox

Web-based collaborative system for supporting online creative collaboration

Focus on the context of animation production

Two modes

Sandbox Planner

Sandbox Improv

Evaluation

Compare and contrast both Sandbox modes with existing leadership practices

Mixed methods

Log files, content analysis, and in-depth interviews

Learn More Research Paper: “Leadership in Online Creative Collaboration” Authors: Kurt Luther and Amy Bruckman URL: http://doi.acm.org/10.1145/1460563.1460619 Course: CS 4690 Empirical Methods for HCI Instructor: Professor Rebecca Grinter URL: http://www.cc.gatech.edu/~beki/cs4690/CS4690_CS6455.html Lecture notes are online My email: [email_address]

Research Paper: “Leadership in Online Creative Collaboration”

Authors: Kurt Luther and Amy Bruckman

URL: http://doi.acm.org/10.1145/1460563.1460619

Course: CS 4690 Empirical Methods for HCI

Instructor: Professor Rebecca Grinter

URL: http://www.cc.gatech.edu/~beki/cs4690/CS4690_CS6455.html

Lecture notes are online

My email: [email_address]