Chapter 8 Sampling Sampling Sampling involves decisions about who or what will be tested, observed, or interviewed in your study (Morse, 2007) Key questions to address: Who should and should not be included? How many should be included? Probability Probability is the likelihood that an event or a condition will occur You can express probability in terms of the chance the event will occur or in percentages Levels of Significance Levels of significance are the difference that will be accepted as too large to be attributed to chance These levels are set by the researcher at the outset of a study Probability Samples Probability samples are formed to ensure that each subject has an equal chance of being included so an unbiased sample can be used Probability Samples A sampling design explains how the subjects are chosen and should include: Number of subjects How they will be assessed, screened, and selected Inclusion and exclusion criteria Probability Samples Random selection is accomplished by having: Identification of all possible participants Every potential participant is given an equal chance of being selected Probability Samples Variations of random sampling include: Stratified: randomly select from each stratum Cluster: sample groups rather than individuals Multistage: sample from multiple sets of clusters Nonprobability Sampling Reasons why researchers use nonprobability samples are: Limited resources for developing an accurate sampling frame or purchase lists of potential subjects Information needed to identify all potential subjects is not available Nonprobability Sampling Reasons why researchers use nonprobability samples are: Limited number of subjects Subjects are difficult to find or difficult to persuade to participate in study Subjects do not complete study Experimental mortality Nonprobability Sampling Types of nonprobability samples include: Quota sampling: select a specified number of participants from each group Convenience sampling: enroll those who are available Snowball network or referral sampling: begin with known individuals and ask them to refer others who meet selection criteria Tracking and Reporting Sample Development In order to improve the reporting of randomized controlled trials (RCTs), the Consolidated Standards of Reporting Trials (CONSORT) were developed A flow diagram that can be used for tracking sample development CONSORT Flow Diagram Source: Altman, D.G., Schulz, K.F., Moher, D., Egger, M.. Davidoff, F., Elbourne, D., Gøtzsche, P.C., & Lang, T. (2001). The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Annuals of Internal Medicine; 134(8), 663-694. Example of Flowchart Source: Buchbinder, R., Osborne, R.H., Ebeling, P. R., Wark, J.D., Mitchell, P.M., Wriedt, C., Graves, S.D., Staples, M.P., & Murphy, B. (2009). A randomized trial of vertebroplasty for painful osteoporotic vertebral factures. The New England Journal of Medicine, 361 ...

1. Chapter 8
Sampling
Sampling
Sampling involves decisions about who or what will be tested,
observed, or interviewed in your study (Morse, 2007)
Key questions to address:
Who should and should not be included?
How many should be included?
Probability
Probability is the likelihood that an event or a condition will
occur
You can express probability in terms of the chance the event
will occur or in percentages
Levels of Significance
Levels of significance are the difference that will be accepted as
too large to be attributed to chance
These levels are set by the researcher at the outset of a study
Probability Samples
Probability samples are formed to ensure that each subject has
an equal chance of being included so an unbiased sample can be
used

2. Probability Samples
A sampling design explains how the subjects are chosen and
should include:
Number of subjects
How they will be assessed, screened, and selected
Inclusion and exclusion criteria
Probability Samples
Random selection is accomplished by having:
Identification of all possible participants
Every potential participant is given an equal chance of being
selected
Probability Samples
Variations of random sampling include:
Stratified: randomly select from each stratum
Cluster: sample groups rather than individuals
Multistage: sample from multiple sets of clusters
Nonprobability Sampling
Reasons why researchers use nonprobability samples are:
Limited resources for developing an accurate sampling frame or
purchase lists of potential subjects
Information needed to identify all potential subjects is not
available
Nonprobability Sampling
Reasons why researchers use nonprobability samples are:
Limited number of subjects

3. Subjects are difficult to find or difficult to persuade to
participate in study
Subjects do not complete study
Experimental mortality
Nonprobability Sampling
Types of nonprobability samples include:
Quota sampling: select a specified number of participants from
each group
Convenience sampling: enroll those who are available
Snowball network or referral sampling: begin with known
individuals and ask them to refer others who meet selection
criteria
Tracking and Reporting
Sample Development
In order to improve the reporting of randomized controlled
trials (RCTs), the Consolidated Standards of Reporting Trials
(CONSORT) were developed
A flow diagram that can be used for tracking sample
development
CONSORT Flow Diagram
Source: Altman, D.G., Schulz, K.F., Moher, D., Egger, M..
Davidoff, F., Elbourne, D., Gøtzsche, P.C., & Lang, T. (2001).
The revised CONSORT statement for reporting randomized
trials: Explanation and elaboration. Annuals of Internal
Medicine; 134(8), 663-694.

4. Example of Flowchart
Source: Buchbinder, R., Osborne, R.H., Ebeling, P. R., Wark,
J.D., Mitchell, P.M., Wriedt, C., Graves, S.D., Staples, M.P., &
Murphy, B. (2009). A randomized trial of vertebroplasty for
painful osteoporotic vertebral factures. The New England
Journal of Medicine, 361(6), 557-568.
Types of Errors in
Quantitative Research
A type I error occurs when a null hypothesis that is true is
rejected
A type II is when we fail to reject a false null hypothesis
Power Analysis Using Effect Size
The power of a statistical test is the probability that it will yield
a statistically significant result
An underpowered study is when the sample is too small and
leads to a type II error
An overpowered study is when the sample is too large and leads
to a type I error
16
Power Analysis Using Effect Size
Effect size is the estimated magnitude of the phenomenon under
study
An effect size calculation indicates the strength of the
relationship between the independent and dependent variables
The equation is: d =( XC – X1) / SD pooled

5. 17
Equivalence of Effect Size to Correlation
CoefficientdrSmall.20.10Medium.50.30Large.80.50
Power Analysis Using Effect Size
If the null hypothesis is not true, then the effect size will be
greater than zero
The larger the effect size, the greater the degree to which the
phenomenon is shown
The larger the effect size is, the greater the power will be so a
smaller sample is needed
Purposeful Sampling
The sampling done for qualitative studies is called purposeful
because it is directed by the purpose of the study, not by
statistical calculations
It is also called purposive, judgmental, or theoretical sampling
20
Types of Sampling Used
in Qualitative Designs
Case study: the case or cases selected are unusual in some way
Ethnography: select a culture, subculture, or ethnic group of
interest; begin with “big net” and then narrow it down

6. 21
Types of Sampling Used
in Qualitative Designs
Phenomenology: select subjects who experienced the
phenomenon under study
Grounded theory: selection of subjects and sources of data
based on their ability to contribute to the evolving theory
22
Sampling Strategies
Extreme cases
Intense experiences
Maximum variation sampling
Negative instances or confirming and disconfirming cases
23
Sampling Strategies
Homogeneous sampling
Criterion sampling
Stratified purposeful sampling

7. 24
Evolving or Iterative Sampling
Reasons why the sampling strategy may be altered during the
study:
Saturation
Scope
Variation
Verification
25
Purposive Sample Size
Single or multiple cases
8 to 12 participants for focus groups
Theoretical saturation
Redundancy
26
Chapter 9
Reliability
What is Reliability?
Reliability is concerned with questions of consistency

8. Other terms for reliability are:
Repeatability
Reproducibility
Stability
Consistency
Predictability
Agreement
Homogeneity
Measurement
Measurement is the assignment of number to object or events
according to certain rules (Carmines and Zeller, 1979)
Measurement
Measurement is important in quantitative research because:
Quantification allows for powerful statistical analysis
Numbers are often more clearly communicated
Objectivity is increased
Efficiency may be increased
Levels of Measurement
Nominal: a label but nothing more
Categorical: identifies group membership
Ordinal: indicates an order
Interval: also in order but an estimation of distance between the
scores
Ratio: order, defined distance, and a zero point
Measurement Error
The sources of error causing unreliability may be one or more
of the following:

9. Measurement is inaccurate or inconsistent
Raters or testers are inaccurate or inconsistent
Measurement Error
The sources of error causing unreliability may be one or more
of the following:
Phenomenon being measured varies from one measurement time
to the next
The situation is confounding the measurement
Classic Measurement EquationX =t
+eObservedTrueRandomScoreScoreError
Consistency
In order to maintain consistency of measurement there needs to
be:
Interrater reliability
Intrarater reliability
Intercoder reliability
Cohen’s Kappa
A way to calculate the percent of agreement between the two
coders
K = fo – fc K = kappa
N – fc fo = frequency of agreement

10. fc = frequency expected by chance
N = number evaluated
Test-Retest Reliability
A type of reliability that is evaluated by administering the same
test to the same people or taking the same measurement on the
same people after a specified period of time
The results of the two testing times are then compared
statistically
Test-Retest Reliability
Factors affecting the test-retest reliability:
Assumes stability in the phenomenon being measured
May be affected by reactivity
Practice effect may also affect reliability
Test-Retest Reliability
Ways to calculate test-retest reliability include:
Pearson product moment correlation
Intraclass correlations (ICCs)

11. Homogeneity
Cronbach’s alpha can be used to test the homogeneity of items
within a measure
It indicates the extent to which all of the items on the test are
“behaving” similarly
Homogeneity
Alpha of 0.70 is acceptable for new measures
Alpha of at least 0.80 is expected for established measures
Higher alphas (at least 0.90 or higher) are desirable for use in
clinical evaluation
Reliability of Physical Measures
Systematic error: a consistent error
Random error: inconsistent, unpredictable errors
Random errors can cancel each other out unless the researcher
know how to detect them by using the technical error of
measurement (TEM)
√ ∑ d2 d = the difference between scores of paired examiners
__________
2 N N = number of pairs of scores
Technical Error of Measurement (TEM)
Improving Reliability

12. Thoroughly trained raters
Periodic monitoring of raters
Retest and calibrate instruments
Add appropriate items and delete those that lower the alpha
coefficient to increase homogeneity
Improving Reliability
Standardize the conditions under which testing is done and
minimize any distractions
Make instructions clear, standardized