Empirical research methods for Information Science

1. IS 4800 Empirical Research Methods
for Information Science
Class Notes March 16, 2012
Instructor: Prof. Carole Hafner, 446 WVH
hafner@ccs.neu.edu Tel: 617-373-5116
Course Web site: www.ccs.neu.edu/course/is4800sp12/

2. Outline
• Sampling and statistics (cont.)
• T test for paired samples
• T test for independent means
• Analysis of Variance
• Two way analysis of Variance

3. 3
Relationship Between Population
and Samples When a Treatment
Had No Effect
Population

M1 M2
Sample 2
Sample 1

4. 4
Relationship Between Population
and Samples When a Treatment
Had An Effect
Control
group
population
c
Control
group
sample
Mc
Treatment
group
sample
Mt
Treatment
group
population
t

5. Population

Mean? Variance?
2

Sampling
Sample of size N
Mean values from all possible
samples of size N
aka “distribution of means” MM = 
N
X
M

=
N
M
2
2 
 =
N
M
X
SD
 
=
2
2
)
(
ZM = ( M -  ) / M


6. Z tests and t-tests
t is like Z:
Z = M - μ /
t = M – μ / μ = 0 for paired samples
We use a stricter criterion (t) instead of Z because is
based on an estimate of the population variance while is
based on a known population variance.
M

M
S
M
S
M

S2 = Σ (X - M)2 = SS
N – 1 N-1
S2
M = S2/N

7. Given info about
population of change
scores and the
sample size we will
be using (N)
T-test with paired samples
Now, given a
particular sample of
change scores of
size N
We can compute the
distribution of means
We compute its mean
and finally determine
the probability that this
mean occurred by
chance
?
 = 0
S2 est 2 from sample = SS/df
M
S
M
t =
df = N-1
S2
M = S2/N

8. t test for independent samples
Given two
samples
Estimate population
variances
(assume same)
Estimate variances
of distributions
of means
Estimate variance
of differences
between means
(mean = 0)
This is now your
comparison distribution

9. Estimating the Population Variance
S2 is an estimate of σ2
S2 = SS/(N-1) for one sample (take sq root for S)
For two independent samples – “pooled estimate”:
S2 = df1/dfTotal * S1
2 + df2/dfTotal * S2
2
dfTotal = df1 + df2 = (N1 -1) + (N2 – 1)
From this calculate variance of sample means: S2
M = S2/N
needed to compute t statistic
S2
difference = S2
Pooled / N1 + S2
Pooled / N2

10. t test for independent samples, continued
This is your
comparison distribution
NOT normal, is a ‘t’
distribution
Shape changes depending on
df
df = (N1 – 1) + (N2 – 1)
Distribution of differences
between means
Compute t = (M1-M2)/SDifference
Determine if beyond cutoff score
for test parameters (df,sig, tails)
from lookup table.

11. ANOVA: When to use
• Categorial IV
numerical DV (same as t-test)
• HOWEVER:
– There are more than 2 levels of IV so:
– (M1 – M2) / Sm won’t work

12. 12
ANOVA Assumptions
• Populations are normal
• Populations have equal variances
• More or less..

13. 13
Basic Logic of ANOVA
• Null hypothesis
– Means of all groups are equal.
• Test: do the means differ more than expected
give the null hypothesis?
• Terminology
– Group = Condition = Cell

14. 14
Accompanying Statistics
• Experimental
– Between-subjects
• Single factor, N-level (for N>2)
– One-way Analysis of Variance (ANOVA)
• Two factor, two-level (or more!)
– Factorial Analysis of Variance
– AKA N-way Analysis of Variance (for N IVs)
– AKA N-factor ANOVA
– Within-subjects
• Repeated-measures ANOVA (not discussed)
– AKA within-subjects ANOVA

15. 15
• The Analysis of Variance is used when you have more
than two groups in an experiment
– The F-ratio is the statistic computed in an Analysis of
Variance and is compared to critical values of F
– The analysis of variance may be used with unequal sample
size (weighted or unweighted means analysis)
– When there are just 2 groups, ANOVA is equivalent to the t
test for independent means
ANOVA: Single factor, N-level
(for N>2)

16. One-Way ANOVA – Assuming
Null Hypothesis is True…
Within-Group Estimate
Of Population Variance
2
1
est

2
2
est

2
3
est

2
est
within

Between-Group Estimate
Of Population Variance
M1
M2
M3
2
est
between

2
2
est
within
est
between
F


=



17. Justification for F statistic

18. Calculating F

19. Example

20. Example

21. Using the F Statistic
• Use a table for F(BDF, WDF)
– And also α
BDF = between-groups degrees of freedom =
number of groups -1
WDF = within-groups degrees of freedom =
Σ df for all groups = N – number of groups

22. One-way ANOVA in SPSS

23. 23
Data
0
1
2
3
4
5
6
1 Day 2 Day 3 Day
Performance
Mean

24. 24
Analyze/Compare Means/One Way
ANOVA…

25. SPSS Results…
ANOVA
Performance
24.813 2 12.406 9.442 .001
27.594 21 1.314
52.406 23
Between Groups
Within Groups
Total
Sum of
Squares df Mean Square F Sig.
F(2,21)=9.442, p<.05

26. 26
Factorial Designs
• Two or more nominal independent variables,
each with two or more levels, and a numeric
dependent variable.
• Factorial ANOVA teases apart the contribution
of each variable separately.
• For N IVs, aka “N-way” ANOVA

27. 27
Factorial Designs
• Adding a second independent variable to a single-
factor design results in a FACTORIAL DESIGN
• Two components can be assessed
– The MAIN EFFECT of each independent variable
• The separate effect of each independent variable
• Analogous to separate experiments involving those variables
– The INTERACTION between independent variables
• When the effect of one independent variable changes over levels of a
second
• Or– when the effect of one variable depends on the level of the other
variable.

28. Example
Wait Time Sign in Student Center
vs.
No Sign
Satisfaction

29. 0
2
4
6
8
10
12
Level 1 Level 2
Level of Independent Variable A
Value
of
the
Dependent
Variable
Level 1 Level 2
Example of An Interaction - Student Center Sign –
2 Genders x 2 Sign Conditions
F
M
No
Sign
Sign

30. 30
Two-way ANOVA in SPSS

31. 31
Analyze/General Linear
Model/Univariate

32. 32
Results
Tests of Between-Subjects Effects
Dependent Variable: Performance
26.507a
5 5.301 3.685 .018
210.855 1 210.855 146.547 .000
20.728 2 10.364 7.203 .005
.002 1 .002 .001 .974
1.680 2 .840 .584 .568
25.899 18 1.439
401.250 24
52.406 23
Source
Corrected Model
Intercept
TrainingDays
Trainer
TrainingDays * Trainer
Error
Total
Corrected Total
Type III Sum
of Squares df Mean Square F Sig.
R Squared = .506 (Adjusted R Squared = .369)
a.

33. 33
Results

34. 34
Degrees of Freedom
• df for between-group variance estimates for main
effects
– Number of levels – 1
• df for between-group variance estimates for
interaction effect
– Total num cells – df for both main effects – 1
– e.g. 2x2 => 4 – (1+1) – 1 = 1
• df for within-group variance estimate
– Sum of df for each cell = N – num cells
• Report: “F(bet-group, within-group)=F, Sig.”

35. Publication format
Tests of Between-Subjects Effects
Dependent Variable: Performance
26.507a
5 5.301 3.685 .018
210.855 1 210.855 146.547 .000
20.728 2 10.364 7.203 .005
.002 1 .002 .001 .974
1.680 2 .840 .584 .568
25.899 18 1.439
401.250 24
52.406 23
Source
Corrected Model
Intercept
TrainingDays
Trainer
TrainingDays * Trainer
Error
Total
Corrected Total
Type III Sum
of Squares df Mean Square F Sig.
R Squared = .506 (Adjusted R Squared = .369)
a.
N=24, 2x3=6 cells => df TrainingDays=2,
df within-group variance=24-6=18
=> F(2,18)=7.20, p<.05

36. 36
Reporting rule
• IF you have a significant interaction
• THEN
– If 2x2 study: do not report main effects, even if
significant
– Else: must look at patterns of means in cells to
determine whether to report main effects or not.

37. Results?
TrainingDays
Trainer
TrainingDays * Trainer
Sig.
0.34
0.12
0.41
n.s.

38. Results?
TrainingDays
Trainer
TrainingDays * Trainer
Sig.
0.34
0.12
0.02
Significant interaction between TrainingDays
And Trainer, F(2,22)=.584, p<.05

39. Results?
TrainingDays
Trainer
TrainingDays * Trainer
Sig.
0.34
0.02
0.41
Main effect of Trainer, F(1,22)=.001, p<.05

40. Results?
TrainingDays
Trainer
TrainingDays * Trainer
Sig.
0.04
0.12
0.01
Significant interaction between TrainingDays
And Trainer, F(2,22)=.584, p<.05
Do not report TrainingDays as significant

41. Results?
TrainingDays
Trainer
TrainingDays * Trainer
Sig.
0.04
0.02
0.41
Main effects for both TrainingDays,
F(2,22)=7.20, p<.05, and Trainer,
F(1,22)=.001, p<.05

42. “Factorial Design”
• Not all cells in your design need to be tested
– But if they are, it is a “full factorial design”, and you
do a “full factorial ANOVA”
Real-Time Retrospective
Agent
Text
 
 X

43. 43
Higher-Order Factorial Designs
• More than two independent variables are included in a
higher-order factorial design
– As factors are added, the complexity of the experimental
design increases
• The number of possible main effects and interactions increases
• The number of subjects required increases
• The volume of materials and amount of time needed to complete the
experiment increases