Kraut/Kiesler Study 1Kraut/Kiesler Study Data Analysis Critique Edgardo Donovan RES 610 – Dr. Joshua Shackman Module 1 – Case Analysis Monday, July 25, 2011
Kraut/Kiesler Study 2 Kraut/Kiesler Study Data Analysis Critique The Kraut/Kiesler study attempted find evidence in favor of a hypothesis positing thatInternet use was negatively correlated to social involvement and psychological well being. Itresulted in a quasi experimental study with an accompanying set of correlated survey data. Aproject focusing on a smaller more defined dependent variable, coupled with more effective useof mediating/moderating variables, and the selection of random survey participants and/ormultiple groups would have had a better chance of leading to quantitative results indicating ahigher degree of external validity. The HomeNet study draws a sample of 93 families from eight diverse neighborhoods inPittsburgh, Pennsylvania. People in these families began using a computer and the Internet athome either in March 1995 or March 1996. Within these 93 families, 256 members signedconsent forms, were given email accounts on the Internet, and logged on at least once. Childrenyounger than 10 and uninterested members of the households are not included in the sample(Kraut 1998, p. 1020). Demographic characteristics, social involvement, and psychological well-being of participants in the HomeNet trial were measured on a pretest questionnaire, before theparticipants were given access to the Internet. After 12 to 24 months, participants completed afollow-up questionnaire containing the measures of social involvement and psychological well-being. During this interval, their Internet usage using custom- designed logging programs wasautomatically recorded. The data reported here encompasses the first 104 weeks of use after aHomeNet familys Internet account was first operational for the 1995 subsample (Kraut 1998, p.1021). To measure family communication, participants were asked to list all the members oftheir household and to estimate the number of minutes they spent each day communicating witheach member (Kraut 1998, p. 1021). The data analysis examined how changes in people’s use of
Kraut/Kiesler Study 3the Internet over 12 to 24 months were associated with changes in their social involvement andpsychological well-being. Their initial levels of social involvement and psychological well-being, as well as certain demographic and control variables were statistically controlled (Kraut1998, p. 1023). The sample examined was selected to be diverse, but it was small and notstatistically representative of a particular geographic region or population (Kraut 1998, p. 1029). In 1998 Kraut utilized a longitudinal study as he attempted to find evidence in favor ofthe hypothesis that use of the Internet destroyed social relationships. The model buildingapproach taken in the study was quasi-experimental. We can classify designs into a simplethreefold classification by asking some key questions. First, does the design use randomassignment to groups? No. Kraut selected a small group of home Internet users based inPennsylvania of varying demographic, professional, and social backgrounds. It almost appearsthat the criterion for selection although not random by virtue of the delimited geographiclocation, was solely willingness to participate. When random assignment is used, we call thedesign a randomized experiment or true experiment. If random assignment is not used, then wehave to ask a second question: Does the design use either multiple groups or multiple waves ofmeasurement? If the answer is yes, we would label it a quasi-experimental design. A number ofthe measures were derived from surveys of the participants, taken at the beginning and again atthe end of the approximately year-long study. Internet use was measured through a monitoringdevice in the home and a series of usage logs that were supposed to be kept by the participants. The research process can be viewed as a series of interlocking choices, in which we trysimultaneously to maximize several conflicting goals. There is not one true method that willguarantee success. A researcher begins by noticing a real world phenomenon and attempts tocreate new knowledge by inquiring about its dynamics. The research problem is then
Kraut/Kiesler Study 4incorporated into a specific design which then in turn gives birth to an operational plan. Thatplan may take on different approaches. In most cases the end-result from the execution of thatplan involves acquiring data that proves correlations or associations among a well defined set ofvariables that support a new way of understanding. Although there are many different ways tosequence research, typically the series of choices is locally directional: plan must come beforeexecution; data collection must come before data analysis (McGrath 180). A better approach for the Kraut/Keisler study would have been to focus on deliveringfurther research perhaps in different geographical areas to in seeking greater external validity ofthe results. That is not necessarily an easy thing to accomplish. A major difficulty in a quasiexperimental approach through a longitudinal design is being able to get people to participateand actually finish the study. It is necessary to strike the right balance when attempting toincentivize a group to participate in a study for conflicts of interest have a strong potential forrendering the results of otherwise well designed research to be invalid. Typically, it is importantfor researchers to be honest about their research goals so as to not mislead and disgruntleparticipants later on who will respond with a lack of seriousness. Sometimes financial incentivesmay succeed in swaying people to participate but will create unnecessary conflicts of interest.The longer a research project is the more fragile it becomes for it is more difficult to ensure thatparticipants participate wholeheartedly over a long period of time. Multiple linear regression is widely used in empirically-based policy analysis. Much of thisuse is inappropriate, not because of the multiple linear regression methodology, but because ofthe nature of the data used. Too often analysts are carried beyond justified inferences intoassertions for which there is essentially no sound defense, leading to policy recommendations ofdubious provenance (Porter 1981, p. 397). The Kraut/Kiesler study does not use random data
Kraut/Kiesler Study 5rather applies statistical controls to help the data fit the hypothesis and over utilizes multipleregressions to show correlations. One of the problems with the research design is some of thevariables such as social involvement and psychological well being are dependent on a myriad ofother variables (diet, personality, stress, rest, emotional challenges, fulfillment in any aspect oflife, etc.) that are not addressed in the questionnaire nor the qualitative portion of the study.Kraut merely started with a database thought to be relevant and then estimated the coefficientsfrom the data base. By examination of the fitted models, predictions were generated as to whatconsequences may be expected to result from policy interventions that alter particular variablevalues (Porter 1981, p. 398). The Kraut/Kiesler study would have been much improved if a moreexhaustive attempt to include more potential independent variables were introduced. However,the overall design was solid in that it included elucidation of hypotheses, followed by testing,and retesting: describe a data set, generate directional hypotheses on what variables predict anoutcome of interest, design a study to test those hypotheses, if confirmed, formulate cause andeffect hypotheses, design a study to test the causal hypotheses by manipulating the causes, andperform successive replications to confirm the hypothesized result (Porter 1981, p. 413). Kraut uses a qualitative literary review to form the basis for a path-analysis approach. Inany given study, the relationship between model and data is strongly conditioned by the priorlevel of knowledge and theory available. When these are advanced, a deductive approach ispossible: strong hypotheses may be formulated initially, and then tested by means of the modelof the data (Porter 1981, p. 399). However, path analysis is merely one of multiple ways thatsequential causal models can be shown. More complex models could certainly be formulated -e.g., path analyses, usually recursive; simultaneous equation models; feedback dynamics models;or other simulation models. Such models rely on deductive reasoning: while they can be tested
Kraut/Kiesler Study 6on a single data base, they cannot sensibly be induced from it (Porter 1981, p. 401).Kraut/Kiesler used path analysis to test the relationships among variables measured at three timeperiods: pretest questionnaire at Time 1 (T1), Internet usage during Time 2 (T2), and posttestquestionnaire at Time 3 (T3). The statistical associations among demographic characteristics,social involvement, and psychological well-being measured at T1 and Internet use measured atT2 provide an estimate of how much preexisting personal characteristic led people to use theInternet (Kraut/Kiesler 1998, p. 1023). Figure 1. Causal Paths (Garson 2008, p. 1) Path analysis is an extension of the regression model, used to test the fit of thecorrelation matrix against two or more causal models which are being compared by theresearcher. The model is usually depicted in a circle-and-arrow figure in which single-headedarrows indicate causation. A regression is done for each variable in the model as a dependent onothers which the model indicates are causes. The regression weights predicted by the model arecompared with the observed correlation matrix for the variables, and a goodness-of-fit statistic iscalculated. The best-fitting of two or more models is selected by the researcher as the best modelfor advancement of theory. Path analysis requires the usual assumptions of regression. It isparticularly sensitive to model specification because failure to include relevant causal variables
Kraut/Kiesler Study 7or inclusion of extraneous variables often substantially affects the path coefficients, which areused to assess the relative importance of various direct and indirect causal paths to the dependentvariable (Garson 2008, p. 1). However, path analysis does not confirm causation in a model.Ultimately path analysis deals with correlation, not causation of variables variable (Garson 2008,p. 1). The strength of a compelling theory will to a large degree persuade whether that correlationis a result of causality. If there is no plausible theory behind the correlation than it may be due tochance. The Kraut/Keisler study formed a plausible theory underlying the relationship betweentwo or more variables then proceeded to test the hypothesis to see if the correlations responded tothe proposed theory. In this study path analysis measured three points in time to denote survey responsesprior to the study (T1), during the study (T2), and upon completion (T3). The goal was to show anegative correlation between social well being and Internet use. Although correlations wereshown causality was not measured. The study featured a number of different kinds of effects ofone variable on another -- direct, indirect, large, small, exogenous, endogenous, mediating, andmoderating. Mediating and moderating variables stand in a model somewhere between theoriginal predictor or predictors (independent variables) and the final criterion or criteria(dependent variables). After research has been designed, money spent in collecting data, and the first analysesare made there must be a great temptation to polish what you have and present it for publication.However, one must truly make an attempt of being a fierce devil’s advocate of one’s research itand try to find its weaknesses in order to strengthen or at least have a better opportunity todeliver better work in the future. Bootstrapping provides the most powerful and reasonablemethod of obtaining confidence limits for specific indirect effects under most conditions.
Kraut/Kiesler Study 8Researchers must also entertain the possibility of multiple mediators. In most situations, it isunlikely that the effect of an independent variable on an outcome is transmitted by only onemeans (Hayes 2008, p. 886) When thinking about a potential research design for my eventual dissertation I will striveto produce a viable experimental design formulating hypotheses inferred from a carefulqualitative analysis of previous recognized academic contributions. The data I will seek will needto be random. The challenge will be to attempt to operationalize an array of variables in such away to maintain a controlled environment rather than simulate one statistically as Kraut/Kiesler.If that is not possible, I will have to settle for a quasi-experimental design by using multiplegroups or multiple groups of measurement. I suspect that I will have to assess the potential effectof many different bi-directionally mediating and moderating variables. If successful in doing so Iwill have a better chance that the results will show a high validity for whatever hypothesis I amtrying to prove by virtue of being meticulous in defining the environment. However, this mayresult in something akin to niche-research in that it may have lower external validity as a result. The Kraut/Kiesler study attempted find evidence in favor of a hypothesis positing thatInternet use was negatively correlated to social involvement and psychological well being. Itresulted in a quasi experimental study with an accompanying set of correlated survey data. Aproject focusing on a smaller more defined dependent variable, coupled with more effective useof mediating/moderating variables, and the selection of random survey participants and/ormultiple groups would have had a better chance of leading to quantitative results indicating ahigher degree of external validity.
Kraut/Kiesler Study 9 BibliographyGarson, D. (2008) Path Analysis. StatNotes. North Carolina State University. RetrievedAugust 24, 2008, fromhttp://www2.chass.ncsu.edu/garson/pa765/path.htmKraut, Robert and others (1998) Internet paradox: a social technology that reduces socialinvolvement and psychological well-being? American Psychologist 53(9):1017–1031McGrath, Joseph E & Brinberg, David (1983). External validity and the researchprocess: a comment on the calder/lynch dialogue. The Journal of Consumer Research.10(1). 115-124.McGrath, Joseph E. (1981) Dilemmatics: The study of research choices and dilemmas.American Behavioral Scientist. 25(2). 179-211.Porter, AL, Connolly, T, Heikes, RG And Park, CY (1981) Misleading indicators: thelimitations of multiple linear regression in formulation of policy recommendations.Policy Sciences 13 (1981) 397-418.Preacher, K. J., & Hayes, A. F. (2008). Asymptotic and resampling strategies forassessing and comparing indirect effects in multiple mediator models. Behavior ResearchMethods, 40, 879-891. Retrieved August 24, 2008, from http://www.comm.ohio-state.edu/ahayes/indirect2.pdf