Critique of an Article QUESTIONS TO ASK WHILE READING A RESEARCH ARTICLE INTRODUCTION – Why did they start? · Why is this research important? How well did the authors justify this in the review of literature? Did they summarize the existing literature relevant to the issue addressed? · What is the research question/s of interest (purpose/objectives) or hypothesis? Is the phrasing comprehensive, clear, specific? · Do the authors use well-defined, non-ambiguous terminology (avoiding jargon)? METHODS – What did they do? · Did the authors provide sufficient detail for the replication of the study? · What’s the study design? Are there any potential threats to the internal and external validity of the design? · Have the data collection procedures and instruments described? What is the dependent variable (DV)? Independent variables (IV)? Is the dependent variable (outcome measure) well defined and clearly measurable? Are the IVs (exposure or condition to be manipulated) well defined/clearly measurable? · Who is the sample population? Does this population place any limits on the generalizability of the results? Was the sampling procedure/subject selection described? · Sample size – How do the authors justify the number of subjects in the study? Are “drop outs” accounted for? · How was their sample assessed? Was the measure validated? · Does the study design have any control condition to rule out chance findings? Are there threats to the internal and/or external validity of the study? · Is the time frame adequate to answer the research question? RESULTS – What did they find? · Were the number of subjects in each group or subgroup used in the analysis specified? Were the subject characteristics summarized? · Did the results reported relate to the specified objective/hypotheses? · Do the tables and figures “speak for themselves?” Are they useful and clear? · Are the tables adequately titled, labeled, and footnoted so they can be interpreted without reference to text? DISCUSSION – What do the results mean? · Did authors discuss the results in relations to the objectives/hypotheses? Is there a clear distinction between the results of the hypothesis being tested and other, unexpected findings being discussed? · Were the results discussed in relation to those from similar studies? Discussed in relation to the theoretical background for this field of study? · Are the authors justified in the strength of the statements they make in the study? Did they offer alternative explanations for results? · How will these results add to this field of research? Were potential directions for future studies mentioned? REFERENCES – Are they comprehensive and current? MISCELLANEOUS Did the article: · Define ambiguous terms? · Define acronyms the first time they appeared in text? · Provide references, citations for statement of facts? · Follow logical sequencing of facts? · Present information clearly and concisely? REMEMBER A ‘critique’ does .

1. Critique of an Article
QUESTIONS TO ASK WHILE READING A RESEARCH
ARTICLE
INTRODUCTION – Why did they start?
· Why is this research important? How well did the authors
justify this in the review of literature? Did they summarize the
existing literature relevant to the issue addressed?
· What is the research question/s of interest
(purpose/objectives) or hypothesis? Is the phrasing
comprehensive, clear, specific?
· Do the authors use well-defined, non-ambiguous terminology
(avoiding jargon)?
METHODS – What did they do?
· Did the authors provide sufficient detail for the replication of
the study?
· What’s the study design? Are there any potential threats to the
internal and external validity of the design?
· Have the data collection procedures and instruments
described? What is the dependent variable (DV)? Independent
variables (IV)? Is the dependent variable (outcome measure)
well defined and clearly measurable? Are the IVs (exposure or
condition to be manipulated) well defined/clearly measurable?
· Who is the sample population? Does this population place any
limits on the generalizability of the results? Was the sampling
procedure/subject selection described?

2. · Sample size – How do the authors justify the number of
subjects in the study? Are “drop outs” accounted for?
· How was their sample assessed? Was the measure validated?
· Does the study design have any control condition to rule out
chance findings? Are there threats to the internal and/or
external validity of the study?
· Is the time frame adequate to answer the research question?
RESULTS – What did they find?
· Were the number of subjects in each group or subgroup used
in the analysis specified? Were the subject characteristics
summarized?
· Did the results reported relate to the specified
objective/hypotheses?
· Do the tables and figures “speak for themselves?” Are they
useful and clear?
· Are the tables adequately titled, labeled, and footnoted so they
can be interpreted without reference to text?
DISCUSSION – What do the results mean?
· Did authors discuss the results in relations to the
objectives/hypotheses? Is there a clear distinction between the
results of the hypothesis being tested and other, unexpected
findings being discussed?
· Were the results discussed in relation to those from similar
studies? Discussed in relation to the theoretical background for
this field of study?
· Are the authors justified in the strength of the statements they

3. make in the study? Did they offer alternative explanations for
results?
· How will these results add to this field of research? Were
potential directions for future studies mentioned?
REFERENCES – Are they comprehensive and current?
MISCELLANEOUS
Did the article:
· Define ambiguous terms?
· Define acronyms the first time they appeared in text?
· Provide references, citations for statement of facts?
· Follow logical sequencing of facts?
· Present information clearly and concisely?
REMEMBER
A ‘critique’ does not mean a negative review. Comment on the
strengths as well as the weaknesses. In critiquing articles, you
should be critical but also creative. Think beyond what is
presented and visualize what issues are not mentioned, what
unknown or unmeasured variable(s) might have resulted in some
other explanation for the outcome.
Not all of these questions apply to all of the articles, nor are
these the only questions to ask about a study. These are just
some ideas to get you started. These questions represent an
approach to critically evaluating the research literature.

4. COMPARISON OF SAMPLING STRATEGIES
Type and Definition
Description of Steps
Advantages
Disadvantages
I. Simple random sampling—when each individual in a defined
population has an equal and independent chance of being
selected into the sample.
1. Assign to each member of population a unique number.
2. Select via use of random numbers (random number table,
dice, computer, etc.) the sample members in a sufficient number
1. Maximum external validity, assuming reasonably small
refusal rate.
2. Requires minimum knowledge of the population
characteristics in advance
3. Free of possible classification errors
4. Very simple to implement
5. Easy to analyze data & compute error
1. Researcher must complete population list (often difficult)
2. Doesn’t use knowledge of population researcher may have
3. For same sample size, produces larger sampling error
compared to stratified random sampling.
II. Systematic random sampling—when each individual in a
defined population has an equal (but not independent) chance of
being selected into the sample
1.Compute sampling interval r=N/n, where
N=number in population

5. n=number needed in sample
round up to an integer
2. Randomly select a start #
3. Select every rth individual
1.Maximum external validity, assuming no ordering in the list
or file of names
2. Very simple/quicker than Simple random sampling because
there is no need for a numbered list
3. Easy to analyze data & compute error
1. If sampling interval is related to a periodic order, increased
variability may be introduced
2. Estimates of errors likely to be high where there is an order
3. May produce errors if N is miscalculated initially.
III. Multistage random sampling—when each individual in
randomly sampled units have an equal chance of being selected
into the sample.
1. Use random sampling (I or II) to select some sampling units
(companies, schools, classes, etc.)
2. Use random sampling (I or II) to select individuals from each
sampling unit.
1.Sampling lists, identification, numbering are required only for
members in sampling units; especially advantageous with large
or difficult-to-enumerate populations
2. If sampling units are geographically defined, this reduces
data collection costs
3. High external validity
1. Sampling error larger than I or II for same sample size
2. Sampling error increases as number of units sampled in first
stage decreases.
IV. Stratified random sampling—

6. a. Proportionate:
when each individual in purposively defined strata has an equal
and independent chance of being selected into the sample
1. Divide population list into strata on the basis of their relevant
characteristic(s)
2. Randomly select from each stratum a number of sample
members proportionate to the size of each stratum
1. Assures representativeness of sample with respect to
stratification variable
2. Decreases chance of failure to have a sufficient number of a
subgroup(s) needed for desired analysis
3. Less extraneous variability than I-III.
4. Medium high external validity
1. Requires accurate information on proportion of population in
each stratum; otherwise, increased error
2. May be costly, time-consuming to achieve stratified
population list
3. Possibility of faulty classification that creates higher random
variance
b. Disproportionate:
when each individual in purposively defined strata has an
unequal but random chance of being selected into the sample
1. Same as IV.a
2. Randomly select from each stratum a number of sample
members disproportionate to the sized of each stratum (i.e., one
or more strata “overrepresented”)
1. More efficient than IV.a for comparing across strata (fewer
total number required)
2. Assures having a sufficient number of a low incidence
subgroup of population
3. Medium external validity

7. 1. Same as IV.a on strata
2. Less efficient than IV.a for point estimates for entire
population
3. Must use sampling weights prior to statistical analysis; make
the data analysis more complex
Type and Definition
Description of Steps
Advantages
Disadvantages
V. Cluster (or area probability) sampling—
a. Simple
when each individual in randomly selected clusters have an
equal and independent chance of being selected into the sample
1. Randomly select clusters or geographical area (e.g., states,
counties, census tracts) by some form of random sampling (I or
II)
2. Include all members of each cluster in sample (i.e.,
enumeration)
1. Has lowest interviewer data collection costs of all probability
sampling methods
2. Requires listing of only individuals within the sampling
clusters (or areas) which reduces time and money costs
3. Characteristics of clusters can also be used in research/data
analysis (or cluster can be used as the unit of analysis)
1. Larger errors for comparable n than other probability samples
2. Requires unique assignment of each individual to exactly one
cluster; inability to do so results in duplication and/or omission
of individuals
3. Medium external validity
b. Stratified:

8. when each individual in randomly selected clusters and
purposively defined strata have an equal and independent
chance of being selected into the sample
1. Divide clusters into strata by stratum characteristics
2. Randomly select clusters from within each stratum
3. Include all members of each cluster in sample (i.e.,
enumeration)
1. Reduced variability compared to V.a; more efficient for
comparison by strata
2. Comes closer than V.a to assuring researcher the ability to
make relevant comparison of clusters across the different strata
1. Disadvantages of stratified added to those of the simple
cluster (compounds distance from simple random sampling)
2. Cluster properties may change after characteristics are
measured
3. Medium to low external validity
-----------------------------------------------
---------------------------------------------
----------------------------------------------
----------------------------------------------
VI. Quota sampling—
When only a predetermined proportion of a population with
only characteristic(s) specified have a chance of being selected
as a subject
1. Classify population members by some relevant variable(s)
2. Determine the proportion of sample desired with relevant
characteristic(s)
3. Fix a quota of subjects with desired characteristic(s) for each
observer/data collector
1. Reduces costs of obtaining sample members, and, perhaps,
data collection
2. May introduces some stratification effect (but researcher
won’t know for sure until after data collection /analysis)

9. 3. If third step is done randomly, this may make it more like
stratified sample (but, still can’t really be sure it is)
1. Variability and bias of estimates can’t be measured or
adjusted for
2. Possible bias of researchers’ misclassification of subjects
3. Introduces biases of nonrandom selection by observers /data
collectors that differ by observer
4. Low external validity
VII. Judgment or purposive sampling—
When only purposively selected individuals have a chance of
being selected as a subject
1. Select subgroup(s) of the defined population than on the basis
of best information is judged to be representative of the target
population
2. Enumerate, select, or recruit individuals from subgroup(s)
1. Reduces costs of obtaining sample members, and, perhaps
data collection since this is typically done where the subgroups
are geographically proximate
2. Quick
1. Variability and bias of estimates can’t be measured or
adjusted for
2. Requires strong assumptions about the population and its
subgroup(s)
3. Violates all assumptions of all statistical techniques
4. Very low external validity
VIII. Convenience/snowball /volunteer sampling—
When individuals become subjects by convenience, referral, or
by volunteering

10. Nor real method. Subjects are selected or recruited for
researchers’ convenience and minimization of cost and/or time
1. Reduces costs of obtaining sample members, and, perhaps,
data collection
2. Very quick
1. Violates all assumptions of all statistical techniques
2. No external validity; very high probability that sample is
NOT representative of any population
Adapted from Ackoff, R. L. (1953). The design of social
research. Chicago: University of Chicago Press.
FCS 681 Lecture 5
Sampling
What is sampling and Why sample?Selecting part of a larger
groupPurpose is to make generalizations about the population
from the smaller group.Why not study everyone?
--Cost: US Decennial Census, 1990, cost $1.5 billion, 2010,
$14 billion
--Quality: When data are collected on a large scale, quality
declines
--Timeliness
Advantages of samplingSelecting less than the entire population
has advantages:
� Less expensive
� Less time

11. � Better quality control
Key Sampling Concepts
Copyright ©2002, William M.K. Trochim, All Rights Reserved
Key sampling conceptsDistinction between the theoretical
population and the actual study populationGeneralizations can
be made only to the actual population
Theoretical Population
The Study or Accessible Population
Theoretical

12. Study
Sampling Frame
The list or procedure defining the POPULATION (From which
the sample will be drawn.)
Examples:
Telephone book
Voters list
Mailing list
Distinguish sampling frame from sample.
Theoretical
Study
Sampling Frame
Selected Sample
Those selected to participate in the study
Not all will necessarily participate
--Non deliverable surveys
--Incomplete surveys
Theoretical

13. Study
Sampling Frame
Selected Sample
Actual
sample
Subjects who complete the study and data is used in data
analysis
Ratio of actual to selected is called response rate
Key Concepts in samplingRandom sampling error
--the difference between the sample results and the results of a
census conducted using identical procedure.Systematic sampling
error
--due to sampling methods or imperfection in execution.
Sampling bias
--biased results due to improper sampling (e.g., a sample of a
population in which some members of the population are less
likely to be included than others)
Sampling bias: an example1936 Presidential election. Literary
Digest poll through telephone incorrectly predicted Alf Landon
the winnerProblems?
--a biased sample
--response bias: respondents were more likely to express their
preference; non-respondents had a different opinion
Rule:If you wish to make inferences to the population from
which the sample was drawn, a Probability sample must be
used.

14. --each element of the population has a known non-zero
probability of selection.
--Randomness is the bases for sample selection
--Tend to give approximate representation of the population
--Allow for prediction of size of error in estimation
Key concepts: sample sizeThe larger the better?The main key is
not size, but representativenessRule of thumb for Quantitative
research
--A minimum of 30 per group
--A minimum of 10 per variable for multiple regressionIn
Qualitative research, it doesn't matter as much
-- Not trying to infer generalizability
Types of Samples
Probability
Non-Probability
Convenience
Purposive
Simple Random
Systematic Random
Multi-stage Random
Stratified Random

15. Cluster Random
Quota
Simple Random SamplingEach element in the population has an
equal AND independent probability of selectionEqual: every
member of a population has an equal chance of being
selected.Independent: Selection of one individual has no
influence on the selection of the next individual.
Randomly select from a numbered
list

16. Systematic Random SamplingEach element has an equal (but
NOT independent) probability of selectionPopulation size N,
desired sample size n, sampling interval r=N/n.Randomly select
a start number and select every rth individualExample: N=64,
n=8, r=64/8=8. Random start number=3.

19. Systematic Random Sampling-cont.Has same error rate as
simple random sample if the list is in random orderProvides the
benefits of implicit stratification if the list is grouped
Systematic Random Sampling-Cont.Runs the risk of error if
periodicity in the list matches the sampling intervalIn this
example, every 4th element is red, and red never gets sampled.
If the random start number had been 4 or 8, ONLY reds would
be sampled.

20. Multi-stage random sampling Each individual in randomly
sampled units have an equal chance of being selected.Use
simple or systematic sampling to select some sample units
(companies, schools, classes, etc.)Use simple or systematic
sampling to select individuals from each sampling unit
Advantageous with large population
Stratified Random SamplingDivide population into groups that
differ in important waysBasis for grouping must be known
before samplingSelect random sample from within each group

22. Stratified Random Sampling-cont.For a given sample size,
reduces error compared to simple random sampling IF the
groups are different from each otherProbabilities of selection
may be different for different groups, as long as they are
knownOversampling small groups improves inter-group
comparisons

24. Random Cluster Sampling-SimplePopulation is divided into
groups (states, counties, census tracts, etc.)Some of the groups
are randomly selectedFor given sample size, a cluster sample
has more error than a simple random sampleCost savings of
clustering may permit larger sample Error is smaller if the
clusters are similar to each other

26. Random Cluster Sampling-SimpleCluster sampling has very
high error if the clusters are different from each otherCluster
sampling is NOT desirable if the clusters are differentIt IS
random sampling: you randomly choose the clustersBut you will
tend to omit some kinds of subjects

28. Random Cluster Sampling-
StratifiedReduce the error in cluster sampling by creating strata
of clustersSample one or more clusters from each stratumThe
cost-savings of clustering with the error reduction of
stratification
Strata

31. Stratification vs. Clustering

32. StratificationDivide population into groups different from each
other: sexes, races, agesSample randomly from each groupLess
error compared to simple randomMore expensive to obtain
stratification information before sampling
ClusteringDivide population into comparable groups: schools,
citiesRandomly sample some of the groupsMore error compared
to simple randomReduces costs to sample only some areas or
organizations
Combines elements of stratification and clustering
• First you define the clusters. Then you group the clusters into
strata of clusters, putting similar clusters together in a stratum.
• Then you randomly pick one (or more) cluster from each of
the strata of clusters
• Then you sample the subjects within the sampled clusters
(either all the subjects, or a simple random sample of them)
Stratified Cluster Sampling
Random Cluster Sampling—
Stratified
Strata

35. Strata

36. The Problem of Non-Response You can randomly select
peopleBut you cannot make people respondNon-response
destroys the generalizeability of the sample. You are
generalizing to people who are willing to respond to surveysIf
response is 90% or so, not so bad. But if it is 50%, this is a
serious problem
The Problem of Non-Response-cont.Multiple call-backs (or
contacts) are essential for trying to reduce non-response
biasSamples without call-backs have high bias: cannot really be
considered random samplesResponse rates have been fallingIt is
very difficult to get above a 60% response rateYou do the best
you can, and try to estimate the effect of the error by getting as
much information as possible about the predictors of non-
response.

37. Non-probability SamplesQuotaJudgment or
PurposiveConvenience
Quota SamplingClassify population members by some relevant
variables (e.g. sex)Determine the proportion of sample desired
with relevant characteristics (e.g., 100 women, 100 men)Fix a
quota of subjects with desired characteristics for each data
collectorNo call-backs or other features to eliminate
convenience factors in sample selection
Quota Vs Stratified SamplingIn Stratified Sampling, selection
of subject is random. Call-backs are used to get that particular
subject.Note: Stratified sampling without call-backs may not, in
practice, be much different from quota sampling.In Quota
Sampling, interviewer selects first available subject who meets
criteria: is a convenience sample.

38. Judgment or Purposive SamplesSubjects selected for a good
reason tied to purposes of research or researcher’s
judgmentUsually Small samples, not large enough for power of
probability sampling.Nature of research requires small
sampleChoose subjects with appropriate variability in what you
are studyingHard-to-get populations that cannot be found
through screening general population
Convenience SampleSubjects selected because it is easy to
access them. No reason tied to purposes of research.Students in
your class, people on campus, friends, friends’ friends No
external validity

39. Something keep in mind for random sampling Heterogeneity:
need larger sample to study more diverse populationDesired
precision: need larger sample to get smaller errorSampling
design: smaller if stratified, larger if clusterNature of analysis:
complex multivariate statistics need larger samples
Sampling in your researchOften a non-random selection of basic
sampling frame (city, organization etc.)Fit between sampling
frame and research goals must be evaluatedSampling frame as a
concept is relevant to all kinds of research Non-probability
sampling means you cannot generalize beyond the
sampleProbability sampling means you can generalize to the
population defined by the sampling frame
Key Sampling Concepts
Copyright ©2002, William M.K.
Trochim
, All Rights Reserved
Types of Samples
Probability
Non-Probability
Convenience
Purposive

40. Simple Random
Systematic Random
Multi-stage Random
Stratified Random
Cluster Random
Quota
Simple Random Sampling
•Each element in the population
has an equal AND independent
probability of selection
•Equal: every member of a
population has an equal chance of
being selected.
•Independent: Selection of one
individual has no influence on the
selection of the next individual.
Randomly select
from a numbered
list
Systematic Random Sampling
•Each element has an equal (but NOT independent) probability
of selection
•Population size N, desired sample
size n, sampling interval r=N/n.
•Randomly select a start number
and select every rthindividual
•Example: N=64, n=8, r=64/8=8.
Random start number=3.
Systematic Random Sampling -cont.
•Has same error rate as simple
random sample if the list is in
random order
•Provides the benefits of implicit
stratification if the list is grouped
Systematic Random Sampling -Cont.
•Runs the risk of error if

41. periodicity in the list matches
the sampling interval
•In this example, every 4
th
element is red, and red never
gets sampled. If the random
start number had been 4 or 8,
ONLY reds would be
sampled.
Stratified Random Sampling
•Divide population into
groups that differ in
important ways
•Basis for grouping must
be known before
sampling
•Select random sample
from within each group
Stratified Random Sampling -cont.
•For a given sample size, reduces error
compared to simple random sampling IF
the groups are different from each other
•Probabilities of selection may be
different for different groups, as long as
they are known
•Oversampling small groups improves
inter-group comparisons
Random Cluster Sampling -Simple
•Population is divided into groups
(states, counties, census tracts, etc.)
•Some of the groups are randomly
selected
•For given sample size, a cluster
sample has more error than a simple
random sample
•Cost savings of clustering may

42. permit larger sample
•Error is smaller if the clusters are
similarto each other
Random Cluster Sampling -Simple
•Cluster sampling has very
high error if the clusters are
different from each other
•Cluster sampling is NOT
desirable if the clusters are
different
•It IS random sampling: you
randomly choose the clusters
•But you will tend to omit
some kinds of subjects
Random Cluster Sampling -
Stratified
•Reduce the error in cluster
sampling by creating strata
of clusters
•Sample one or more
clusters from each stratum
•The cost-savings of
clustering with the error
reduction of stratification
Strata
Stratification vs. Clustering
Stratification
•Divide population into
groups different from each
other: sexes, races, ages
•Sample randomly from
each group
•Less error compared to
simple random
•More expensive to obtain
stratification information

43. before sampling
Clustering
•Divide population into
comparable groups:
schools, cities
•Randomly sample some of
the groups
•More error compared to
simple random
•Reduces costs to sample
only some areas or
organizations
Stratified Cluster Sampling
Strata
Random Cluster Sampling —
Stratified
The Problem of Non -Response
•You can randomly select people
•But you cannot make people respond
•Non-response destroys the generalizeabilityof
the sample. You are generalizing to people who
are willing to respond to surveys
•If response is 90% or so, not so bad. But if it is
50%, this is a serious problem
The Problem of Non -Response-cont.
•Multiple call-backs (or contacts) are essential for
trying to reduce non -response bias
•Samples without call -backs have high bias: cannot
really be considered random samples
•Response rates have been falling
•It is very difficult to get above a 60% response rate
•You do the best you can, and try to estimate the
effect of the error by getting as much information as
possible about the predictors of non -response.
Non-probability Samples
•Quota

44. •Judgment or Purposive
•Convenience
Quota Sampling
•Classify population members by some relevant
variables (e.g. sex)
•Determine the proportion of sample desired with
relevant characteristics (e.g., 100 women, 100
men)
•Fix a quota of subjects with desired
characteristics for each data collector
•No call-backs or other features to eliminate
convenience factors in sample selection
Quota Vs Stratified Sampling
•In Stratified Sampling,
selection of subject is
random. Call-backs are
used to get that particular
subject.
•Note: Stratified sampling
without call-backs may not,
in practice, be much different
from quota sampling.
•In Quota Sampling,
interviewer selects first
available subject who
meets criteria: is a
convenience sample .
Judgment or Purposive Samples
•Subjects selected for a good reason tied to
purposes of research or researcher ’s judgment
•Usually Smallsamples,not large enough for
power of probability sampling.
–Nature of research requires small sample
–Choose subjects with appropriate variability in what
you are studying
•Hard-to-get populations that cannot be found

45. through screening general population
Convenience Sample
•Subjects selected because it is easy to access
them.
•No reason tied to purposes of research.
•Students in your class, people on campus,
friends, friends’friends
•No external validity
Something keep in mind for random
sampling
•Heterogeneity: need larger sample to study more
diverse population
•Desired precision: need larger sample to get
smaller error
•Sampling design: smaller if stratified, larger if
cluster
•Nature of analysis: complex multivariate
statistics need larger samples
Sampling in your research
•Often a non-random selection of basic sampling
frame (city, organization etc.)
•Fit between sampling frame and research goals
must be evaluated
•Sampling frame as a concept is relevant to all
kinds of research
•Non-probability sampling means you cannot
generalize beyond the sample
•Probability sampling means you can generalize to
the population defined by the sampling frame