dkNET Webinar: Choosing Sample Sizes for Multilevel and Longitudinal Studies Analyzed with Linear Mixed Models Presenter: Kylie K. Harrall, MS, Research Instructor, Lifecourse Epidemiology of Adiposity and Diabetes (LEAD) Center, University of Colorado Anschutz Medical Campus Abstract Planning a reproducible study requires selecting a sample size which will ensure appropriate statistical power. Free point-and-click software (Kreidler et al., Journal of Statistical Software, 2013, 10.18637/jss.v054.i10) makes it easy to select a sample size for clustered and longitudinal designs with linear mixed models. The software, a suite of training modules, and reference materials are freely available online (www.SampleSizeShop.org ). The software interface and training materials are aimed at biomedical scientists, included those funded by NIDDK. We give examples of study designs for which the software will compute power and sample size, including a study with clustering, a study with longitudinal repeated measures, and a study with multiple outcomes, where heterogeneity of response among subgroups is of interest. The top 3 key questions that the Sample Size Shop can answer: 1. What free, online, point-and-click, wizard-style, NIH-funded, validated, published power and sample size software provides calculations for studies with clusters, longitudinal studies, and longitudinal studies with clusters? 2. Can GLIMMPSE (www.SampleSizeShop.org) compute power and sample size for randomized controlled clinical trials and observational studies funded by NIDDK? 3. Why use validated power and sample size software instead of writing simulations? Upcoming webinars schedule: https://dknet.org/about/webinar

1. Kylie K. Harrall, M.S.
Research Instructor
Lifecourse Epidemiology of Adiposity and Diabetes (LEAD) Center – D. Dabelea, Director
University of Colorado – Anschutz Medical Campus
Choosing Sample Sizes for
Multilevel and Longitudinal Studies
Analyzed with Linear Mixed Models
1

2. CONFLICTS OF INTEREST
The authors declare no conflicts of interest.
2

3. CO-AUTHORS
DEBORAH H. GLUECK, PH.D.
DEPARTMENT OF PEDIATRICS
UNIVERSITY OF COLORADO SCHOOL OF MEDICINE
AND
KEITH E. MULLER, PH.D.
DEPARTMENT OF HEALTH OUTCOMES AND BIOMEDICAL INFORMATICS
UNIVERSITY OF FLORIDA COLLEGE OF MEDICINE
3

4. OTHER TEAM MEMBERS
• The version of the software that will be discussed was
primarily developed by Alasdair Macleod, Elizabeth
Litkowski, Qian Li, and Dr. Jiang Bian. Many others
created previous versions (most notably Dr. Sarah
Kreidler).
• The educational technology that will be discussed was
primarily developed by Dr. Albert Ritzhaupt, Dr. Max
Sommer, Dr. Natercia Lourinho Moura do Valle, and
Michelle Zamperlini. Many others helped create the
original course content (most notably Jessica R. Shaw).
4

5. ACKNOWLEDGEMENT OF
SUPPORT
• R01-GM121081, Methods and Software for Lifecourse
Epidemiology Data and Sample Size Analysis, NIGMS,
Glueck, Muller, and Dabelea MPI, 08/16 – 12/22.
• R25-GM111901, A Master Course on Power for Multilevel
and Longitudinal Health Behavior Studies, OBSSR and
NIGMS, Muller and Glueck MPI, 08/14 - 06/18.
5

6. OUTLINE
1. Importance of power analysis
2. Choosing a power analysis design
3. The software
4. Three examples of study designs
5. How to write a power analysis
6. Future directions
6

7. OUTLINE
1. Importance of power analysis
2. Choosing a power analysis design
3. The software
4. Three examples of study designs
5. How to write a power analysis
6. Future directions
7

8. ETHICS OF SAMPLE SIZE
• Scientific knowledge is public and reproducible.
• If the sample size is too small, a study may be inconclusive and waste
resources.
• If the sample size is too large, then a study may expose too many
participants to possible harm.
• Ethical designs have a sample size that is large enough and not too large.
8

9. OUTLINE
1. Importance of power analysis
2. Power analysis approach
3. The software
4. Three examples of study designs
5. How to write a power analysis
6. Future directions
9

10. CHOOSING THE
CORRECT SAMPLE SIZE
REQUIRES ALIGNMENT
• Conscientious study planners routinely align
data analysis with study design and
study design with scientific goals.
• Aligning power analysis with data analysis is not routine because
it requires:
statistical theory,
accurate computational forms, and
easy to use software.
• Until recently, linear mixed models lacked all three.
10

11. POWER THEORY FOR MANY
LINEAR MIXED MODEL
HYPOTHESES
• The core statistical theory of power analysis for a broad swath
of general linear mixed models used in health research is in
Muller et al. (1992) and Chi et al. (2019).
• References in the bibliography give a path to the underlying
theory and the equally important computational methods.
11

12. WHY NOT RUN SIMULATIONS?
12
• Computer simulations take longer because they take time to:
write and debug the simulation,
validate the simulation,
iteratively change the design, and
generate enough points for a power curve.
• Single programmers with a deadline are vulnerable to:
mistakes in initial assumptions,
mistakes in coding,
mistakes in modules written by others and
an inability to validate with independent target values.

13. OUTLINE
1. Importance of power analysis
2. Power analysis approach
3. The software
4. Three examples of study designs
5. How to write a power analysis
6. Future directions
13

14. 14
POWER AND SAMPLE SIZE FOR
MULTILEVEL AND LONGITUDINAL DESIGNS
• GLIMMPSE
• www.SampleSizeShop.org

15. WE GIVE YOU
EASY-TO-USE
AND
VALIDATED
POWER
SOFTWARE
• We have built a point-and-click
application for web browsers.
• The wizard-style software is free, and
copyleft under the GNU public license so
that it will always be free.
• A sequence of dialog boxes leads a user
through well-defined steps.
• The software is validated to industry
standards.
• Software development was funded by five
NIH grants to our group.
15

16. FEATURES OF
GLIMMPSE
SOFTWARE
• Free access is available to anyone.
• A user only needs a web browser.
• No programming expertise is needed.
• No mathematical notation is used.
• The software works on computers, tablets
and smartphones.
• The computations are done on a remote
server.
• A partial or complete study design can be
saved for later use.
16

17. 17
OUR TRACK
RECORD
Users cited our software in 315
peer-reviewed articles since
2013, with exponential growth.
Our software was cited in
manuscripts which also cited
over $150,000,000 in NIH grants.
Users citing our software are
funded by 12 NIH Institutes and
Centers.

18. GLIMMPSE
GIVES POWER
AND SAMPLE
SIZE FOR MANY
DESIGNS
18
• Cluster- or group-randomized trials
• Randomized block (randomized complete block)
designs
• Multilevel observational or randomized designs
• Observational studies with clustering
• Studies with both clustering or grouping and
repeated measures
• Multivariate designs

19. FAIR TRAINING MATERIALS AT
WWW.SAMPLESIZESHOP.ORG
• A free short course on Coursera: Power and Sample Size
for Multilevel and Longitudinal Study Designs.
• The course was designed for scientists.
• The material was presented as a two-credit course.
• The course materials (~1,000 slides, homework exercises,
answers, screenshots, and video lectures) are available
at “Sample Size Workshop” menu item on
SampleSizeShop.org.
• Tutorials, applied articles, and theory articles are also
linked.
19

20. OUTLINE
1. Importance of power analysis
2. Power analysis approach
3. The software
4. Three examples of study designs
5. How to write a power analysis
6. Future directions
20

21. Single level clustering
Time-by-Treatment interaction
Subgroup analysis
THREE
EXAMPLES OF
STUDY
DESIGNS
21

22. SINGLE LEVEL
CLUSTERING
22

23. SINGLE LEVEL CLUSTERING
• An example study is the National Diabetes Prevention Program –
Flex (NDPP-Flex) cluster randomized trial (N. Ritchie, PI).
• Classes of participants will be randomized to either a standard
National Diabetes Prevention Program, or to a National Diabetes
Prevention program class, with flexible goal setting.
23

24. PROPOSED
RANDOMIZED
TRIAL:
NDPP-FLEX
(N. RITCHIE, PI).
24
Recruit & collect baseline data
(HbA1c)
NDPP-Flex:
Patient-centered
goal setting
Standard NDPP:
Pre-set weight loss
& activity goals
25 group sessions 25 group sessions
Collect 12-month
data (HbA1c)
Collect 12-month
data (HbA1c)
Participants
assigned to classes

25. PROPOSED
RANDOMIZED
TRIAL
25
Recruit & collect baseline data
(HbA1c)
NDPP-Flex:
Patient-centered
goal setting
Standard NDPP:
Pre-set weight loss
& activity goals
25 group sessions 25 group sessions
Collect 12-month
data (HbA1c)
Collect 12-month
data (HbA1c)
Participants
assigned to classes
Unit of
clustering

26. SINGLE LEVEL CLUSTERING
• An example study is the National Diabetes Prevention Program –
Flex (NDPP-Flex) cluster randomized trial (N. Ritchie, PI).
• Classes of participants will be randomized to either a standard
National Diabetes Prevention Program, or to a National Diabetes
Prevention program class, with flexible goal setting.
• Participants in the same class will have correlated outcomes.
26

27. PROPOSED
RANDOMIZED
TRIAL
27
Recruit & collect baseline data
(HbA1c)
NDPP-Flex:
Patient-centered
goal setting
Standard NDPP:
Pre-set weight loss
& activity goals
25 group sessions 25 group sessions
Collect 12-month
data (HbA1c)
Collect 12-month
data (HbA1c)
Participants
assigned to classes
Outcome at
12-months
Outcome
at baseline

28. HYPOTHESIZED
DIFFERENCE IN
HBA1C
28
Control
Treatment
Δ
HbA1c

29. SINGLE LEVEL CLUSTERING
• A detailed example of a power analysis for a
similar study (drinking reduction) may be found
on in module 1 on samplesizeshop.org.
29

30. TIME-BY-TREATMENT
INTERACTION
30

31. TIME-BY-TREATMENT
INTERACTION
• An example study is the Community Activation for
Prevention Study (CAPS) a longitudinal randomized trial (J.
Litt, PI).
31

32. 32
LONGITUDINAL RANDOMIZED TRIAL

33. 33
LONGITUDINAL RANDOMIZED TRIAL
Baseline Harvest Winter
Time

34. 34
LONGITUDINAL RANDOMIZED TRIAL
Treatment

35. HYPOTHESIZED
TIME BY
TREATMENT
INTERACTION
35

36. TIME-BY-TREATMENT
INTERACTION
• A detailed example of a power analysis for a
similar hypothesis of time by treatment
interaction in a study without clustering
(memory of pain across time) may be found in
module 2 on samplesizeshop.org.
36

37. SUBGROUP ANALYSIS FOR
LONGITUDINAL STUDY
37

38. SUBGROUP ANALYSIS FOR
LONGITUDINAL STUDY
• Often in a grant proposal, the researcher wishes to assess
subgroup differences in an outcome. For example, the
researcher might want to test if the following groups
have different mean outcomes:
• Sex assigned at birth
• Rural vs. Urban
• Those with mutation versus wildtype
38

39. SUBGROUP ANALYSIS FOR
LONGITUDINAL STUDY
• A longitudinal observational study (Healthy Start, D.
Dabelea, PI) utilized medical records collected during
early childhood.
• The outcome was repeated measures of BMI.
• Up to 20 measurements of BMI were extracted from
medical records.
• The goal was to determine whether the association
between neonatal adiposity and early childhood body
mass index trajectories differed by sex assigned at birth.
39

40. 40
HYPOTHESIZED DIFFERENCE IN BMI
TRAJECTORY BY SEX ASSIGNED AT BIRTH
Longitudinal

41. 41
HYPOTHESIZED DIFFERENCE IN BMI
TRAJECTORY BY SEX ASSIGNED AT BIRTH
Subgroup

42. SUBGROUP ANALYSIS FOR
LONGITUDINAL STUDY
• A detailed example of a power analysis for
a similar study (gene by environment
interaction) may be found on in module 5
on samplesizeshop.org.
42

43. OUTLINE
1. Importance of power analysis
2. Power analysis approach
3. The software
4. Three examples of study designs
5. How to write a power analysis
6. Future directions
43

44. 44

45. A POWER
ANALYSIS
CHECK-LIST
45

46. OUTLINE
1. Importance of power analysis
2. Power analysis approach
3. The software
4. Three examples of study designs
5. How to write a power analysis
6. Future directions
46

47. MISSING DATA
• With no missing data, the mixed models of interest give a
Wald statistic that is equivalent to the Hotelling-Lawley trace.
• With no missing data, the power results are exact in many
important cases.
47

48. WHAT IS NOT AVAILABLE
• GLIMMPSE software covers only a subset of mixed models:
incomplete designs are not allowed.
• Continuous predictors are limited to a single covariate.
• Unequal cluster sizes are not covered.
• Binary, Poisson and survival outcomes are not allowed.
• High dimensional problems are ignored.
• No power software covers everything.
• There is much room for future methodological research.
• There is much room for extensions to the software.
48

49. WE PLAN TO ADD
WHAT USERS WANT
• The preferences of users will guide us. If you have questions or
run into issues, please email us at samplesizeshop@gmail.com.
• We are seeking funding to continue the project.
49

50. BIBLIOGRAPHY
• Chi YY, Glueck DH and Muller KE (2019) Power and sample size for fixed-effects inference in
reversible linear mixed models, The American Statistician, 73, 350-359. doi:
10.1080/00031305.2017.1415972, PMCID: PMC7009022.
• Gedney JJ, Logan H, Baron RS. (2003) Predictors of short-term and long-term memory of sensory
and affective dimensions of pain. J Pain. Mar;4(2):47-55. doi: 10.1054/jpai.2003.3. PMID:
14622715.
• Harrall KK, Muller KE, Starling AP, et al. Power and sample size analysis for longitudinal mixed
models of health in populations exposed to environmental contaminants: a tutorial. BMC Med Res
Methodol. 2023;23(1):12. doi:10.1186/s12874-022-01819-y
• Kenward, M. G. and Roger, J. H. (1997) Small sample inference for fixed effects from restricted
maximum likelihood, Biometrics, 53, 983–997.
• Kreidler SM, Muller KE, Grunwald GK, Ringham BM, Coker-Dukowitz ZT, Sakhadeo UR, Barón AE,
and Glueck DH (2013) GLIMMPSE: online power computation for linear models with and without a
baseline covariate. Journal of Statistical Software, 54(10). PMCID: PMC3882200
• Litt JS, Alaimo K, Harrall KK, et al. Effects of a community gardening intervention on diet, physical
activity, and anthropometry outcomes in the USA (CAPS): an observer-blind, randomised
controlled trial. Lancet Planet Health. 2023;7(1):e23-e32. doi:10.1016/S2542-5196(22)00303-5
• Logan HL, Baron RS, Kohout F. (1995) Sensory focus as therapeutic treatments for acute pain.
Psychosom Med., 57(5):475–484.
• Moore BF, Harrall KK, Sauder KA, Glueck DH, Dabelea D. Neonatal Adiposity and Childhood Obesity.
Pediatrics. 2020;146(3):e20200737. doi:10.1542/peds.2020-0737
• Muller KE, LaVange LM, Ramey SL, and Ramey CT (1992) Power calculations for general linear
multivariate models including repeated measures applications. Journal of the American Statistical
Association, 87, 1209-1226. PMCID: PMC4002049
• Muller KE and Stewart PW (2006) Linear Model Theory; Univariate, Multivariate, and Mixed
Models. New York: Wiley. DOI: 10.1002/0470052147
• Sommer M, Ritzhaupt AD, Muller KE and Glueck DH (2019) Transformation of a face-to-face
workshop into a Massive Open Online Course (MOOC): a design and development case, Journal of
Formative Design in Learning, 3, 97–110. doi:10.1007/s41686-019-00037-y
50
Also see
https://www.SampleSizeShop.org

51. QUESTIONS?
51