This document discusses three potential methods for evaluating the effectiveness of a new educational module (PEM) study: 1. Historical control method compares pre- and post-test scores before and after module implementation but has delayed rollout and is quasi-experimental. 2. Randomized control group method assigns participants randomly to control or intervention groups but requires randomization and potential denial of modules. 3. Geographic control method uses a separate control institution but cannot guarantee similarities between institutions. It allows full rollout without randomization within each institution.

1. A graphical representation of the IRB
methods for the PEM Module study
(PRELIM)
Todd Chang, MD
2012.02.14

2. Timeline
Mar 2012 May July Jan 2013
Apr June
SITE exam

3. Method 1 (historical control)
Mar 2012 May July Jan 2013
Apr June
SITE exam
pretest
posttest
mods
mods
mods
mods
mods
mods
Etc.
Control
Intervention (modules)

4. Method 1 (historical control
PRO CON
• Institutional consistency • Delayed module rollout
• No randomization required • Quasi-experimental,
confounded by what part of
academic year

5. Method 2 (randomized control group)
Mar 2012 May July Jan 2013
Apr June
SITE exam
Control
mods mods
mods mods
mods mods
mods mods
Intervention (modules) mods mods
mods mods

6. Method 2 (randomized control group)
PRO CON
• Large span of institutions • Randomization required
will insure similar traits • Denial of modules to certain
between intervention & groups within institution
control – IRB issues?
– Difficulty with full denial
particularly late in year

7. Method 3 (geographic control)
Mar 2012 May July Jan 2013
Apr June
SITE exam
Control
mods mods
mods mods
mods mods
mods mods
Intervention (modules) mods mods
mods mods

8. Method 3 (geographic control)
PRO CON
• No randomization within • Cannot guarantee
institution similarities between
– Institutional consistency institutions
• Full academic year worth of
‘control data’