The document discusses experiment-free research methods for studying online creative collaborations. It describes conducting naturalistic observation and ethnography of an online animation community. The researcher interviewed collaboration leaders to understand challenges of structuring projects, directing artists, and integrating work. Findings showed few collaborations were completed due to difficulties with design, management, and assembly of collaborative animations. Specifications and leadership style affected collaboration success.

1. Non-Experimental Methods Kurt Luther

2. Non-Experimental Methods Kurt Luther Experiment-Free

3. Non-Experimental vs. Experiment-Free Childless vs. Childfree Not worse, just different!

4. Why Experiment-Free? Some variables can’t be manipulated… Ethically Practically Period! Experiments often based on experiment-free research Exploratory studies Help identify and scope research questions

5. Characteristics of Experiment-Free Research

6. Experiment-Free Research Methods Naturalistic Observation Ethnography Sociometry The Case History Archival Research Content Analysis

7. Experiment-Free Research Methods Naturalistic Observation Ethnography Sociometry The Case History Archival Research Content Analysis

8. Naturalistic Observation vs. Ethnography Naturalistic Observation Unobtrusive observations of subjects’ naturally occurring behavior are made Ethnography The researcher becomes immersed in the behavioral or social system being studied. May be conducted as a participant or non-participant observation study

9. How to Observe What to record Space, time, materials, names Keep research questions in mind Don’t constrain too much at beginning When to record In-vivo vs. afterwards Recording equipment Notepad Audio recorder Digital camera, video camera Types of notes Jottings Observable events, conversations, etc. Analyses of what you’ve seen Diary Emotional responses

10. Jottings vs. Diary From Jules Henry, Culture Against Man (1963) "Boris had trouble reducing 12 over 16 to the lowest terms and could only get as far as 6 over 8. The teacher asked him quietly if that was as far as he could reduce it. She suggested he 'think'. Much heaving up and down and waving of hands by the other children, all frantic to correct him. Boris pretty unhappy, probably mentally paralyzed. The teacher, quiet, patient ignores the others and with look and voice concentrates on Boris. She says, ' Is there a bigger number than two you can divide into the two parts of the fraction?' After a minute or two, she becomes more urgent, but there is no response from Boris. She then turns to the class and says, ' Well, who can tell Boris what the number is?' A forest of hands appears, and the teacher calls Peggy. Peggy says that four may be divided into the numerator and the denominator.” “ Boris' failure has made it possible for Peggy to succeed; his depression is the price of her exhilaration. This is the standard condition of American elementary schools…so often somebody's success is bought at the cost of our failure. To a Zuni, Hopi or Dakota Indian Peggy's performance would be viewed as cruel beyond belief, for competition, the writing of success from somebody’s failure, is a form of torture foreign to those non-competitive [cultures]. Yet Peggy’s action seems natural to us; and so it is. How else would you run the world? And since all but the brightest have the constant experience that others succeed at their expense they cannot but develop an inherent tendency to hate -- to hate the success of others, to hate the successful and become determined to prevent it. Along with this, naturally, goes the hope that others will fail…”

11. History of Ethnography 19 th Century Origin in anthropology Western Expansionism North America: Boas and Native Americans Europe: Colonial encounters Mostly “armchair anthropology” 20 th and 21 st Century Malinowski and Trobriand Islands Sociology and Chicago School Looking inward Marginalized populations Prostitutes, prisoners, homeless, mentally ill Modern ethnography Subcultures Technological adoption Within organizations Virtual ethnography (online communities)

13. “ Pass-my-Flash 2” An example collaborative animation project (collab) http://www.newgrounds.com/portal/view/428554

14. Some Numbers Year created: 2008 Audience views: 65,000+ Participants: 7 Ages: 17 to 29 Countries: 2 U.S. and CAN Cities: 5 Brooklyn, Ontario, Orlando, Savannah, Washington D.C.

15. Some Questions How did these animators meet ? How did they agree on a collab? How was the work divided up ? How much planning was involved? What roles , if any, did animators adopt? What other types of collabs exist? How does all of this compare with other types of online, collaboratively-created projects?

16. Study Sites Three online animation communities Primarily Newgrounds.com Created in 1995 by Tom Fulp 1,500,000+ registered members 130,000+ member-contributed animations Two other “satellite communities” < 500 members Contribute to Newgrounds.com Scope Focused on movie collabs

17. Research Questions How do people collaborate over the Internet to create animated short movies? ( i.e. , what is the “collab production process”?) What challenges must leaders manage throughout the collab production process and how do leaders manage these challenges? How might technology be designed to help collab leaders manage these challenges?

20. Interviewing Technically, in-depth, phenomenological interviewing Why? Understand experiences of collab leaders Descriptive account of collab production process Recruitment Purposeful sampling (collab participants) Via discussion forums, private messages, “snowball sampling” Participants 14 telephone, 3 email Ages: 16 to 29; all male 6 countries (USA, UK, Australia, Spain, Netherlands, Estonia) Real names vs. pseudonyms [Bruckman 2002] Questions Start with generic interview guide Semi-structured format

21. Participant Demographics * Led at least one collab † Pseudonym Name Age Country Joseph Blanchette * 24 U.S. Eric Carlson 19 U.S. Luis Castanon * 27 U.S. Michael Frank * 19 U.S. Tom Fulp * 29 U.S. James Hole * 16 Australia Tyler Koch * 19 U.S. Massimo Maitan * 21 Australia Anders-Martin Meister 16 Estonia Ross O’Donovan * 19 Australia Kraig Phillips 27 U.S. Joseph Rooks * 21 U.S. Kester Smith 21 U.K. Anonymous 1 † 18 Netherlands Hans Van Harken * 17 Spain Robert Westgate 21 U.K. Anonymous 2 † 19 U.S.

23. > > Hello there, my name is Dan, I'm 14, and I just found a thread on a > > forum I go to on a regular basis, and the person who posted it > > claims to be a part of GVU, and it doing interviews on &quot;Creative > > Collaborations in Online Communities&quot;, and he wants to do all the > > interviews over the phone. Naturally, I am a little skeptical on > > whether or not this is a scam or not. I checked through the names > > of the students currently in GVU, but he wasn't there. He claims to > > be a Mr. Kurt Luther, and if you want, here's the link to the thread: > > http://ngcollabs.galacti.com/bbs/viewtopic.php?t=2586 > > > > there's a few reasons why I'm a bit worried about it: > > 1) As I said earlier, I couldn't find him in the enlisted students. > > 2) He won't do interviews over Instant Messanging sevices, and > > insists they be over the phone > > 3) almost none of the people at NG Collabs are over 18 > > 4) the field he is reasearching is so limited > > 5) we are a very small community, with only about 50 active members > > 6) www.newgrounds.com/bbs has a much bigger forum for flash > > collaborations, and would get much more response, not to mention > > one of the only ways to get to NG Collabs is through Newgrounds, > > seeing as we are trying to have our site hosted by them > > > > So I am very worried about this, and so I wanted some proof that > > Kurt Luther actually does exist. > > If you could get back to me sometime on this matter, I would > > appreciate it. > > > > Thank you for your time, > > Dan

27. Grounded Theory Analysis Structured approach to analyzing qualitative data Often used for analysis of interview data Allows themes to emerge from the data Bottom-up coding Open coding Label phenomena (concepts) Discover categories (groups of concepts) Axial coding Make connections between categories Selective coding Select the core category (story line) Relate other categories to the story line Result Series of interrelated themes telling one coherent story

28. Questions? Acknowledgments Interviewees, Newgrounds.com community, and Tom Fulp Anonymous CSCW 2008 reviewers ELC Lab, especially Sarita Yardi Kelly Caine, Pam Griffith, Beki Grinter, and Shruthi Panicker, Kevin Ziegler National Science Foundation You (thanks for listening!) Contact us Kurt Luther [email_address] Amy Bruckman [email_address]

29. Ethnography Issues Observing as a participant or non-participant Gaining access to a field setting Gaining entry into the group Becoming invisible vs. “going native” Making observations and recording data Analyzing ethnographic data

30. Experiment-Free Research Methods Naturalistic Observation Ethnography Sociometry The Case History Archival Research Content Analysis

31. Content Analysis Basics Used to analyze a written or spoken record for occurrence of specific behaviors or events Archival sources often used as sources for data Appears simple, but may be complex Should be used within a clearly developed study, including hypotheses to be tested Response categories must be clearly defined A method for quantifying behavior must be defined

32. Performing a Content Analysis Clearly defined response categories are essential Two units of analysis Recording unit: Element of the material you are going to record (e.g., instances of a certain word) Context unit: Context within which material analyzed appears Observers doing content analysis must be blind so that bias will not enter the analysis Materials to be analyzed should be chosen carefully to increase generality Cannot be used to establish causal connections among variables

33. Content Analysis of Collab Threads Why? How many collabs result in a completed animation? How many do not? Why? Content All collab activity is kept public Collab threads “ Screen scraping” via Python script/MySQL database ~ 1,600 between Sept. 2003 to Sept. 2006 Criteria Complete or incomplete? Why? Coding Performed manually by two judges Calculated inter-rater reliability

37. Advanced Content Analysis Detailed content analysis of collab threads “ Screen scraped” all Newgrounds.com collab threads 150,000+ posts total Manually coding a random sample Why? Determine completion rates for different types of collabs Attributes of completed vs. incomplete collabs What does a “likely to succeed” collab look like? Leadership style, specs, arrangements, themes, etc. Understand how collab structures change over time

39. Findings Few collabs (<20%) are completed 3 major challenges for leaders Designing the project Structuring, proposing Managing the artists Recruiting, directing, motivating, replacing Completing the project Integrating, publishing

40. Findings Few collabs (<20%) are completed 3 major challenges for leaders Designing the project Structuring , proposing Managing the artists Recruiting, directing , motivating, replacing Completing the project Integrating , publishing

41. Structuring the Project Modularization and granularity [Parnas 1972; Benkler 2006] “ Every author on Newgrounds likes to make their own thing. You wouldn’t be able to give one person a job of storyboarding and one person a job of animating it and one person a job of recording the sound. It just wouldn’t work. People have to do their own thing on Newgrounds, so it’s a lot easier to just let them do their piece of animation and take a whole month to do it.” (Massimo Maitan)

42. Structuring the Project Specs Technical guidelines e.g. , frame rate, dimensions Themes Content guidelines e.g. , narrative, music, visual element, event, vignette Arrangements Collaboration guidelines Linear, continuous, nonlinear “ [Y]ou have to give people really concrete boundaries in terms of how to put their movie together—not the creative part, but the technical aspects of it—in order for it to succeed to begin with.” (Luis Castanon) “ If people are more free, people will come up with a huge range of ideas, and people won’t get bored over the course of the Flash. But at the same time, they’ve got to be restrictive enough so that when the Flash is put together, it works.” (Robert Westgate)

45. Structuring the Project Specs Technical guidelines e.g. , frame rate, dimensions Themes Content guidelines e.g. , narrative, music, visual element, event, vignette Arrangements Collaboration guidelines Linear, continuous, nonlinear “ [Y]ou have to give people really concrete boundaries in terms of how to put their movie together—not the creative part, but the technical aspects of it—in order for it to succeed to begin with.” (Luis Castanon) “ If people are more free, people will come up with a huge range of ideas, and people won’t get bored over the course of the Flash. But at the same time, they’ve got to be restrictive enough so that when the Flash is put together, it works.” (Robert Westgate)

46. “ Valentine ‘29” Linear Story Collab

47. “ Pass-my-Flash 2” Continuous Story Collab

48. “ When Farm Animals Attack” Nonlinear Story Collab

49. More Examples Concept Example Arrangement Theme Theme Title Yr. # Parts. # Auths. Participant(s) Linear Narrative chapter Events of St. Valentine’s Day Massacre “ Valentine ‘29”† 2007 6 6 Hans Van Harken* Music lyrics “ One O’Clock Jump” by Tommy and Jimmy Dorsey “ One O’Clock Jump”† 2005 11 9 Luis Castanon* “ Stop Crying Your Heart Out” by Oasis “ Stop Cryin Your Heart Out”† 2005 24 4 Luis Castanon “ We Hate When Our Friends Become Successful” by Morrissey “ We Hate It When…”† 2005 16 5 Ross O’Donovan Genre selection Television series episode “ The Clockcrew TV Collab 2”† 2007 50+ 1 Joseph Rooks* Improvisational Continuous visual element Red line “ The Red Line”† 2007 50+ 1 Robert Westgate Mountain slope “ Mount Newgrounds” 2007 90+ 2 Robert Westgate Nonlinear Vignette prompt Farm animals attacking people “ The ‘W.F.A.A.’ Collab” 2006 10 5 Massimo Maitan* Pirates or ninjas “ NG TimeTrial IX”† 2004 4 4 Luis Castanon* Emoticons “‘ Emoticonisation’ Collab”† 2006 12 5 Massimo Maitan* Make fun of Mr_Artist “ The Mr. Artist Collab” 2005 5 4 Michael Frank* Cooking “ Cooking with Clockcrew”† 2006 7 7 Tyler Koch* Pop culture parody Transformers film parodies “ Blamformers!”† 2007 30+ 9 Tom Fulp* The Matrix film parodies “ The Matrix Still Has You”† 2004 5 5 Joe Blanchette* Halo 2 videogame parodies “ TSAH”† 2004 6 5 Ross O’Donovan* Halo 2 videogame parodies “ TSAH2”† 2006 5 5 Joe Blanchette Format/style constraint Pixilated graphics “ Retro Collab”† 2007 14 10 James Hole* Frame-by-frame animation style “ NG TT Series Finale”† 2005 7 7 Luis Castanon* Time constraint Complete animation within 3 days “ NG Time Trial Challenge”† 2004 10 4 Luis Castanon*

50. Directing the Artists Everyone’s a volunteer Authorial leadership Balance authority and egalitarianism Too much Artists drop out Too little Under-utilization Collab never completed Commitment Leaders can’t quit Creative vision doesn’t transfer Logistical issues “ If you’re collaborating, you gotta make everybody feel like they’re a part of it. You’ve got to make them feel like it’s all their movie. Because if it’s not, then they won’t want to work on it.” (Tyler Koch) “ There have been times where people have said, ‘Yeah, you’re too bossy.’ And I’m like, ‘Yeah, but if I’m not bossy, you guys are never gonna get it done.’” (Ross O’Donovan) “ I don’t think of it as a position of power. I think of it as a position that enables me to…give them things to participate in.” (Joseph Rooks)

51. Integrating the Animations Recomposition Integration challenges Aesthetic Variety vs. continuity Social Ownership Technical Compilation issues File size Flash symbols “ [Variety is] one of the best parts about collaborations, different people’s art styles coming together…but it’s still important [that], like, in some way it flows.” (James Hole) “ How would you feel if someone…changed your work without telling you? It’s just…you should at least inform the person.” (Anders-Martin Meister) “ The hard thing for me, working on [collabs], is dealing with other people’s techniques and methods for making stuff.” (Tom Fulp)

53. Contributions An empirically-grounded description of practices surrounding online creative collaboration in the open-ended problem domain of entertainment A characterization of three major challenges faced by online creative collaboration leaders while creating entertainment and how they manage these challenges. A discussion of these challenges vis-à-vis those faced by leaders of other forms of online creative collaboration

54. What’s Next New tool: Sandbox Web-based collaborative system for supporting online creative collaboration Focus on the context of animation production Two modes Sandbox Planner Sandbox Improv Evaluation Compare and contrast both Sandbox modes with existing leadership practices Mixed methods Log files, content analysis, and in-depth interviews

55. Learn More Research Paper: “Leadership in Online Creative Collaboration” Authors: Kurt Luther and Amy Bruckman URL: http://doi.acm.org/10.1145/1460563.1460619 Course: CS 4690 Empirical Methods for HCI Instructor: Professor Rebecca Grinter URL: http://www.cc.gatech.edu/~beki/cs4690/CS4690_CS6455.html Lecture notes are online My email: [email_address]