Methodology and IRB/URR
Dr. James Lani
Take Away Message
Research design is a blueprint with
• Research method: Quant/Qual/Mixed
• Operational constructs: How
constructs are measured
• Sampling strategy and procedure
What is Research
Research follows the scientific method.
What’s the scientific method?
• Formulation of testable questions
• It’s organized knowledge: logical
(theory) and evidence based
• Precise constructs
• Can be disproven (falsifiable)
• Parsimonious (simplest
Methodology: The Cookbook Metaphor
Making a stew… Examining research questions…
Ingredients: 2 lbs beef, I clove garlic…
Preparation: Cube beef, mince garlic…
Cooking Instructions: Bake at 350 for one
Celebrate: You’ve made a replicable) stew!
Ingredients: 20 participants, 15 item
questionnaire, semi-structured interview.
Preparation: Administer questionnaire before
and after lecture; semi-structure interview with
participants for 20 minutes using a tape
Data analysis plan: Conduct dependent sample
t-test; transcribe interviews then thematize
Celebrate: You’ve conducted a (replicable)
Where the recipe can be replicated
Methodology Essential Ingredients
Methods Quantitative Qualitative Mixed
You can count it Not quantitative Both
Tend to answer “What
questions” (What is
differences) or “When
questions” (when is
Tend to answer “Why
why people feel that way)
or “How questions”
(explore how they see
Strategies of data
assignment) and Non-
(no control group),
interviews, Archive data,
Observations (write down
positive and negative
feeling words)… Both
Sampling strategy Discuss sampling process Discuss sampling process
Theoretical (or Conceptual) Framework
Theory is a systematic explanation of behavior
• Theory guides analyses
• List existing theories and how your
research questions relate to those
• E.g., Theory of mind: the ability to understand that
others have their own beliefs, desires, intentions.
Empathy. Tested by Faux Pas Task (ability to
recognize a faux pas). Research question: Does
alcohol abuse impact empathy?
The population is the group you want to
• Describe characteristics of population
• Why is population relevant to problem
(look at other peer reviewed study’s
• Distinguish the population from the sample
Sampling Frame and Sample
from universe you
have access to or
Old (one’s own SE) Change in item New (perceptions
of others’ SE)
I feel that I have a
number of good qualities.
“I have” to “she has” I feel that she has a
number of good
I feel I do not have much
to be proud of.
“I do not” to “she
I feel she does not
have much to be
On the whole, I am
satisfied with myself.
“I am” to “she is” and
“myself” to “herself”
On the whole, she is
satisfied with herself.
If you need to amend instrument, use a change
matrix; do not create your own instrument!
Materials: Informed Consent
• State purpose of project
• State procedure and how long it will take
• State voluntary nature of participation
• State risks (if any)
• Have them sign or state that by filing our
survey they are agreeing to participate.
Constructs vs. Variables
Constructs vs. Variables
Constructs are the invisible abstract things we’re measuring (e.g.,
personality), while variables are the way we’re assessing
(measure/operationalize) those invisible things.
Constructs and variables need to be precise (is personality
measured by introversion scale or by conscientiousness scale?)
e.g., Intelligence is a construct, while the number of words
remembered is a way of assessing intelligence.
e.g., Personality is a construct, while the scores on an
introversion/extroversion test is a way to assess an aspect of
Operationalize Constructs, Make Distinction between
IV’s/DV’s, and Describe Level of Measurement
Does Empathy differ by group (alcohol abuse vs. no alcohol abuse)?
Empathy is my dependent variable and Group is my independent
Empathy (my construct) is measured by scores on the Faux Pas task.
Alcohol abuse (my construct) is measured by 5 or more drinks in one
Empathy is a ratio-level variable measured with scores ranging from 0-
25, while my Group variable is a nominal-level (categorical-level) variable
because participants re in one of two groups (alcohol abuse group or
not in alcohol abuse group).
Reliability and Validity
Valid but not
• Internal Validity: IV causes a change in the DV (not time or
other covariates, etc.
• External validity: can be generalized to the population
• Construct validity: does the scale measure the theoretical
• Translational validity:
• Face validity (items are reasonable)
• Content validity (items match the domains of interest)
• Criterion related validity: measures behave as theory predicts
• Convergent validity: how close the variable aligns with the
construct (use EFA)
• Concurrent validity: construct relates to established
• Predictive validity: measure can predict an outcome
Internal consistency: Cronbach alpha. Average
Inter-rater: if interval, correlate; if
Test-retest: administer same test at two times
Split half: divide instrument into 2 parts and
calculated totals, then correlate totals.
Brief Review: Units of Analyses
Quantitative (e.g., Age)
Nominal-level (Latin for name). Gender (M/F), Grouped (Old =
65+, middle age = 36-64, young = 35 or younger). Assign any
number of groups (old = 1, middle = 2, young = 3).
Ordinal-level is ranked (Latin for showing order). GPA (A-F), or
age (group 1 = age 1–15, group 2 = age 16-25, group 3 = age 25-54,
group 4 = age 65+)
Interval/Ratio-level (also named scale or continuous; Latin for
[equal] space [between numbers]). What is your age today in
years? ____ (a number from 1-105)
Types of Methodology Models
Theories explain phenomena,
Models represent Phenomena.
• SEM and Path models
• Regression models (linear, logistic, ordinal)
• ANOVA models (repeated-measures)
• Heirarchical Linear Models (HLM)
• Correlational Models
Relationship Among Variables
Data Collection Method --
• Describe the procedures used to administer
the materials to the participants
• Remember to be as detailed as necessary
so someone can literally replicate your
• Procedure for accessing participants
• Selection of data collected
• Number and duration of interviews
• How and when data is collected
• How data is recorded (e.g., hand
• Role of researcher-relationship to researcher
• Procedure to administer measures
Pilot Test (only if you develop
• Detect potential issues in the instrument
• Allows you to get feedback and to finalize
your survey/interview items
• Makes sure participants understand survey
• Assess typical responses to survey items or
interview questions (were participants
comfortable, long-winded, defensive, etc.
• Can you access data?
Data Analysis Plan
• Quantitative: Describe the analysis plan used
to test each hypothesis, the assumptions of
the statistical analyses, and a justification of
the appropriateness of the analysis for each
• Qualitative: Describe how the data will be
analyzed (or thematized)
• Case Study
• Grounded theory
Varies by type of
1. Research questions
in statistical language
2. Level of
4. Conduct power
Varies by which
• Case Study
• Grounded theory
Assumptions, Limitation, Delimitations
Assumptions: Discuss things out of your control
about the population and design, then justify
assumptions (e.g., participants will answer
Limitations: Are out of your control and
describe weaknesses in design, threats to
validity (e.g., generalizability).
Delimitations: Are in your control and relate to
choices you will make to narrow the scope of
the study (e.g., variables, research questions).
• Describe informed consent procedures
• State whether your study will be anonymous
or confidential with respect to the participants
• Describe considerations for children or