Review of "Research Methods & Design in Psychology"

Lecture 12 Survey Research & Design in Psychology James Neill, 2009 Review & Summary

Overview

Review

Aims and Outcomes
Research process
Survey design
MLR, ANOVA, Power
What type of analysis?

Lab report
Final exam
Evaluation & feedback

Aims & Outcomes

Aims & outcomes 1

Knowledge & skills for conducting ethical , well-designed , survey-based research in psychology.

Aims & outcomes 2

Theory & practice of survey-based research, including:

Research questions / hypotheses
Survey design
Sampling
Interpreting & communicating results

Aims & outcomes 3

Use of SPSS for:

Data entry
Correlations
Factor analysis
Qualitative analysis
Reliability
MLR
Advanced ANOVA

Research Process

The research process 1. Establish need for info/ research 2. Problem definition/ Hypotheses 3. Research design 4. Sampling/ Data collection 5. Data analysis 6. Reporting

Funnel model

Survey Design

Operationalisation of fuzzy concepts
Question types
Response formats
Levels of measurement
Reliability & validity
Sampling
Modes of administration

Items should measure different aspects of the latent construct Latent Construct Measured Construct

Latent Construct Poor items will create ‘brown sludge’ (noisy measure) Measured Construct

Describing Data

Describing data

Data screening
Discrete data:

Frequencies & %s

Continuous data:

4 moments of a normal distribution

Central tendency
Dispersion
Skewness
Kurtosis

Visualisation of data

Aids interpretation of descriptives and tests of differences or relationships.
Univariate :

histogram, bar graph, error-bar graph

Bivariate :

scatterplot, clustered bar graph

Multivariate :

venn diagrams, multiple line graph, 3-d scatterplot

Software for data visualisation (graphing)

Statistical packages

e.g., SPSS Graphs or via Analyses

Spreadsheet packages

e.g., MS Excel

Word-processors

e.g., MS Word – Insert – Object – Micrograph Graph Chart

Factor Analysis

Factor analysis 1

Purpose

Data reduction
Developing reliable & valid measures of fuzzy construct

Assumptions

Linear relations
Sample size

Factor analysis 2

Extraction

PC vs. PAF

Rotation method

Varimax vs. Oblimin

Number of factors

Kaiser’s criterion
Scree plot
Theoretical structure

Factor analysis 3

Decide how many factors
Refine factors and items

Primary loadings > .5?
Cross-loadings < .3?
Sufficient items per factor
Face validity

Correlations between factors
Compare models across groups

Reliabilities & composite scores

Internal reliability: Cronbach’ s  :

> .8 = v.good
> .7 = good
> .6 = OK

Composite scores

Unit-weighting
Regression-weighting

Qualitative Analysis

Qualitative

Purposes

Pilot study, exploratory research
Data reduction -> develop meaning
Validity-testing

Methods

Content analysis
Multiple response analysis
Graph (e.g., bar graph)

Inferential Statistical Decision Making Tree

Statis tical decision tree

Establish the RQ/Hypothesis

Differences or relationships?
No. of IVs and Dvs
Identify levels of measurement

See also:

Inferential statistics decision-making tree

Correlations

Strength & direction of bivariate linear relations
Building block for FA & MLR
Non-parametric correlations
Scatterplots – watch out for:

Outliers
Non-linearity

Caution with causal interpretation

Multiple Linear Regression

Multiple linear regression 1

L inear regression

Y = ax + b + e

Proportion of variance in a continuous DV explained by one or more IVs

R
R 2
Adjusted R 2

Assumptions:

LOM

Continuous DV
Dichotomous or continuous IVs

Normality, linearity & homoscedasticity
Multicollinearity
MVOs

Methods

Standard / Direct
Hierarchical
Stepwise, Forward, Backward

Overall hypothesis:

(Null) That the IVs do not explain variance in the DV (i.e., that R is 0)

One hypothesis per predictor:

(Null) That each IV is not a significant predictor of variance in the DV (i.e., that t for each predictor is less than the critical value)

Also consider:

Direction
Which predictors are more important?
Where IVs are correlated, interpret zero-order vs. partial correlations.

Can use Venn or path diagrams to depict relationships between variables

ANOVA

ANOVA 1

Extension of t -test
ANOVA is like MLR in that there is:

One continuous DV (although ANOVA can handle multiple DVs)
One or more IVs

ANOVA differs from MLR in that:

Interactions are automatically tested
IVs must be categorical
Significant results may indicate need for followup or post-hoc tests

Types of ANOVA

1-way ANOVA
1-way repeated measures ANOVA
2-way factorial ANOVA
Mixed design ANOVA (Split-plot ANOVA)
ANCOVA
MANOVA

ANOVA 3

Assumptions

Cell size > 20 (Ideal)
Normally distributed DVs
Homogeneity of Variance (b/w subjects)
Sphericity (w/in subjects)

Post-hoc and follow-up tests
Calculating η 2 and Cohen’s d

Power and Effect Sizes

Power, effect sizes, significance testing

Significance testing has dominated psychology, but is problematic, mainly because:

Results are dichotomous (sig. or not), which doesn't help us to understand the size of effect.
Sig. test results are influenced by power esp. if particularly high or low.

Power and effect sizes have been neglected. It is good to also:

Calculate the powe r of studies (prospectively &/or retrospectively)
Report ESs & CIs to complement inferential statistics

r, r 2 , R , R 2
η 2 , η p 2 , d

Academic integrity - “Integrity is doing the right thing, especially when no one is watching”.
Publication bias (low power; favouritism of sig. findings)

Lab Report

Lab report - Tips

Check the marking criteria
Use example HD lab report and sample write-ups as guides
Demonstrate your own capability and independent thinking/work
Tell a story

Lab report - Introduction

Explain the rationale – what is the problem of interest?
No waffle – cut to the chase – review literature or argument relevant to the RQs / hypotheses
State clear RQs and/or hypotheses

One per test/analysis/effect

Lab report - Method

3 sections:

Participants
Materials or Instrumentation
Procedure

Efficient & well-organised (like a recipe)
A naïve reader must be able to replicate the study
Include relevant details, but do so efficiently (avoid getting bogged down in extraneous detail)

Lab report – Results 1

Data screening
Consider LOM assumptions
Caution in use of overall scores

Overall Score not valid Overall Score valid 1 3 2 1 3 2

Example MLR: Hierarchical:

DV = Campus Satisfaction
Step 1

IV1 = Gender (M / F)

Step 2

IV1 = IM (Continuous)
IV2 = EM (Continuous)

Table of correlations and regression coefficients

Example ANOVA: 2 x (3) Mixed ANOVA

between-subjects IV: Enrolment Status (FT / PT)
within-subjects DV: Satisfaction (Educational / Social / Campus)

Table of cell and marginal descriptives (M, SD, SK, Kurt) + Graph

Lab report - Discussion

Draw conclusions about the RQs & hypotheses in light of the results.
Discuss key strengths & limitations of the study. (Balanced criticism)

Lab report - Discussion

Make specific, practical, insightful recommendations with regard to theory, research methods, and practice e.g.,

Theory : What are the implications for the theory/rationale upon which the study was based?
Methods : How can the instrumentation be improved?
Practice : What recommend to students and universities e.g., for improving satisfaction?

Lab report – Appendices

Optional: Include where relevant and referred to in the body. Appendices may not be consulted by a reader, so if its important make sure its covered in the text .
Use for content which would break the flow, but which is relevant to understanding the study e.g., the EFA correlation matrix.

Lab report – Appendices

Does not need to be in APA style.
Add labels (e.g., Appendix A, B, C etc.) and possibly names e.g., Appendix A: Bivariate correlations amongst the university student satisfaction items)

Lab report - Submission

Insert and complete the downloadable Cover Sheet

Remember to also include your own Title Page

Submit ONE DOCUMENT containing the coversheet, lab report, appendices
Upload via Moodle

Lab report - Marking

I will endeavour to release lab report marks before the exam , but this will be touch and go.
Please don't ask – I will announce news about lab report marking via Moodle Announcements.

Final Exam

Final exam

120 multi-choice qns in 150 mins. (Mid-semester was 60 qns in 90 mins)
Breakdown:

50 – MLR;
50 – ANOVA;
20 – Sig. testing, power, effect sizes etc.

Practice exam questions come from the same test bank

Final exam – Permitted materials

Non-programmable dictionary
Non-annotated foreign language dictionary
4 sides of A4 notes (Two A4 pages with notes on both sides). Any format – typed and/or written

Evaluation & Feedback

Evaluation & feedback