This document discusses methods for assessing randomization in clinical trials that use covariate-adaptive randomization designs. It presents metrics for measuring randomness, balance, and efficiency loss in randomization schemes. The document outlines a simulation approach and discusses results from comparing different randomization factors and sample sizes. It proposes future directions such as optimizing randomization parameters and exploring periodicity in system behavior.

1. © 2013 Medidata Solutions, Inc. 1© 2013 Medidata Solutions, Inc. 1
Dennis Sweitzer
Principal Biostatistician
Slides & supplementary material wil be posted at:
www.Dennis-Sweitzer.com
7 August 2013, JSM
Randomization Metrics:
Jointly assessing predictability and efficiency
loss in covariate adaptive randomization
designs

2. © 2013 Medidata Solutions, Inc. 2© 2013 Medidata Solutions, Inc. 2
Outline
§ Objective
What is the right questions, anyway?
§ Randomness
How to measure & from whose perspective?
§ Balance
Why? How to measure to match?
§ Simulation
Validity?
§ Results
NB: Slides & supplementary material will be posted at:
www.Dennis-Sweitzer.com blog.mdsol.com

3. © 2013 Medidata Solutions, Inc. 3© 2013 Medidata Solutions, Inc. 3
Questions
Q: Will unequal subgroups affect randomization
performance?
Q: What are the impacts of choosing dynamic
allocation over permuted block?
Q: Dynamic allocation is more deterministic than
permuted block, isn’t it?
Q: What about randomization performance
at interim analysis?
What is the best method for randomizing
THIS study design in
THIS population of patients?

4. © 2013 Medidata Solutions, Inc. 4© 2013 Medidata Solutions, Inc. 4
Measuring Randomness
Randomness
Predictability
(by observer)
Entropy
(no observer)
Periodicity
(patterns)
X ⟶
⟶
Y (as function of probabilities)

5. © 2013 Medidata Solutions, Inc. 5© 2013 Medidata Solutions, Inc. 5
Measuring Balance
Balance
Efficiency
(Variability)
Confounding
(Bias)
Deviation
from Target
(Convenience)
X (as # or % of subjects) ⟶
⟶
Y (as function of probabilities)

6. © 2013 Medidata Solutions, Inc. 6© 2013 Medidata Solutions, Inc. 6
Generic Simulation
Convenient covariates…
Covariate Levels Ratio
2 sexes {M,F} 50:50
3 age groups {Mid,Yng,Old} 33:33:33
10 sites /
variants
{a,b,…,j} 10 x 20 each
However
➣ Real-life trials are
rarely so neat
Although
➣ Simulated trials
usually this neat
pk ∝
1
k +c( )
a
2 sexes 67:33
3 age groups 55:27:18
10 sites 34 : 17 : 11 : 9 : ... : 3.4
Use…
Zipf-Mandelbrot Distribution
➣ Sizes of cities, frequencies of words, species abundance, Website hits…
Q: Will unequal subgroups affect performance?
ANCOVA Model
Outcome = Treatment + Sex + Age + Sex*Age + Variant/Site+error

7. © 2013 Medidata Solutions, Inc. 7© 2013 Medidata Solutions, Inc. 7
Balance: Confounding
Ad hoc:
Score ≣
Total of
#subjects
in covariate
subgroups
with 100%
of a single
treatment
ANCOVA Model
Outcome = Treatment + Sex + Age + Sex*Age + Variant/Site+ error
0 2 4 6 8
CR
Stratified ………...
PB(1:1)
PB(2:2)
PB(4:4)
DAS(0%)
DAS(15%)
Marginal …………
DAM(0%)
DAM(15%)
Strata + Margins
DAE(0%)
DAE(15%)
Site +
DAC(0%)
DAC(15%)
Site+Strata
DAD(0%)
DAD(15%)
Equally
Distributed
Covariates
Zipf-
Mandelbrot
Covariates
Confounding score⟶
A: Yes,
increased
confounding!

8. © 2013 Medidata Solutions, Inc. 8© 2013 Medidata Solutions, Inc. 8
Balance: Loss of Efficiency
ANCOVA Model
Outcome = Treatment + Sex + Age + Sex*Age + Variant/site+ error
0 2 4 6 8 10 12 14 16
CR
Stratified ………...
PB(1:1)
PB(4:4)
DAS(0%)
DAS(15%)
Marginal + ….
DAM(0%)
DAE(0%)
Site + Margins……
DAC(0%)
DAC(15%)
Site+Strata+Margin….
DAD(0%)
DAD(15%)
Loss of Efficiency (LOE) ⟶
Equally Distributed
Covariates
Zipf-Mandelbrot
Covariates
! ! = !! + !!!!
(Atkinson, 2003)
Matrix Form of model,
where:
z ≣treatment allocation
α ≣treatment effect
β ≣Covariate effects
X ≣ Design Matrix
Columns ó Covariates
Rows ó Subjects
Loss of Efficiency:
Var( ˆα) =
σ 2
zt
z−zt
X(Xt
X)−1
Xt
z
LOE = zt
X(Xt
X)−1
Xt
z
A: But not
efficiency

9. © 2013 Medidata Solutions, Inc. 9© 2013 Medidata Solutions, Inc. 9
Randomness: Predictability
Blackwell-Hodges (1957)
guessing rule
☞ Game theory interpretation
☞ Always guesses the next
assignment will restore balance
Measures
Potential Selection Bias
F ≣ abs(# Correct – Expected #
Correct by chance alone)
PotentialSelectionBias(Strata)
Q: Impacts of choosing dynamic allocation
over permuted block ?

10. © 2013 Medidata Solutions, Inc. 10© 2013 Medidata Solutions, Inc. 10
Randomness: PredictabilityPotentialSelectionBias(Strata)
A: More efficiency, less predictability
Randomization factors
Pb ≣ Sex*Age
daC ≣ Sex + Age + Variant
daD ≣ Sex + Age + Sex*Age +Variant

11. © 2013 Medidata Solutions, Inc. 11© 2013 Medidata Solutions, Inc. 11
Randomness: PredictabilityPotentialSelectionBias(Site)
Randomization factors
Pb ≣ Sex*Age
daC ≣ Sex + Age + Site
daD= Sex + Age + Sex*Age +Site
A: … unless the observer knows too much…

12. © 2013 Medidata Solutions, Inc. 12© 2013 Medidata Solutions, Inc. 12
Randomness: PredictabilityPotentialSelectionBias(Site)
Randomization factors
daD= Sex + Age + Sex*Age + Site
daE ≣ Sex + Age + Sex*Age
da? = Sex + Age + Sex*Age + ½ Site
A: However, can adjust weights

13. © 2013 Medidata Solutions, Inc. 13© 2013 Medidata Solutions, Inc. 13
Randomness: Entropy/Syntropy
Observed Entropy
≣ Self Information Content
Where: pj ≣ probability of observed
treatment choice for patient j
Syntropy*
• Average & Rescale to [0,1] so that:
0 ⟹ Max Randomness
1 ⟹ Max Determinism
I = − log(pj )∑
Syntropy
* “Syntropy” ― coined by
Buckminster Fuller as the
opposite of entropy
Q: Isn’t DA deterministic?
A: A random element makes
it as random as PB

14. © 2013 Medidata Solutions, Inc. 14© 2013 Medidata Solutions, Inc. 14
Results: Metrics and Changing Sample SizePotentialSelectionBias(Strata)
Q: What about randomization performance
at interim analysis?
A: PB
becomes more
predictable & a
little more
efficient
A: DAC is and becomes
both less predictable
and more efficient

15. © 2013 Medidata Solutions, Inc. 15© 2013 Medidata Solutions, Inc. 15
Next Directions
§ Compare more methods: Urn randomization, Optimal-Designs,
Novel methods, etc
§ Randomization Metrics vs statistical properties of analyses
§ Optimizing parameters & tweaking algorithms
§ Refining metrics (e.g., Deviation from Target, Periodicity)
§ Exploring quirks in system behavior.
§ For more information (slides, bibliography, supplemental material,
etc.) see:
blog.mdsol.com
OR
www.Dennis-Sweitzer.com
OR
www.slideshare.net/denswei

16. © 2013 Medidata Solutions, Inc. 16© 2013 Medidata Solutions, Inc. 16
Additional Slides
§ Bibliography
§ Randomization factors used
§ Comparing Methods Example
§ Periodicity Plot

17. © 2013 Medidata Solutions, Inc. 17© 2013 Medidata Solutions, Inc. 17
Bibliography
§ Atkinson, AC. (2003) The distribution of loss in two-treatment
biased-coin designs. Biostatistics, 2003, 4, 2, pp. 179–193
§ Blackwell, D. and J.Hodges Jr (1957). Design for the control of
selection bias. Ann Math Stat 28, 449-460
§ Wikipedia contributors. "Entropy (information theory)."
Wikipedia, The Free Encyclopedia. Wikipedia, The Free
Encyclopedia, 23 Apr. 2013. Web. 14 May. 2013.
§ Lebowitsch, J, et al, (2012). “Generalized multidimensional
dynamic allocation method”. Statistics in Medicine,2012;

18. © 2013 Medidata Solutions, Inc. 18© 2013 Medidata Solutions, Inc. 18
Covariates vs Randomization Factors
Analysis: ANCOVA Model
Outcome
= Treatment + Site + Sex + Age + Sex*Age
Stratification factors in Randomization
Strata Imbalances– within combinations of Sex & Age
Marginal Imbalances – within each Sex, Age, and Site
“S”
PB, daS
“M”
daM
“C”
daC
“D”
daD

19. © 2013 Medidata Solutions, Inc. 19© 2013 Medidata Solutions, Inc. 19
Comparing Methods & Parameters
Predictability
vs Loss of Efficiency,
(n=50)
Not much difference in CI
Variations on DA:
• daJS, daJM –
(Kuznetsova, 2012)
• mmS, mmM, baM, baF,
baS - experimental
PotentialSelectionBias(Strata)

20. © 2013 Medidata Solutions, Inc. 20© 2013 Medidata Solutions, Inc. 20
Randomness: Periodicity
A la Discrete Fourier Transform
• Amplitude of a periodic variation in
the max{pi,j} of treatment
assignments
Periodicity