I gave this keynote lecture at the ESSIC 2019 conference. I was specifically asked to share my ideas regarding useful and valid study designs that can convince coverage-decision makers to allow further research on the topic of rare diseases (in this situation Bladder Pain Syndrome with hunner leasions).

2. Complying with regulations concerning the
transparency of event funding and to guarantee
accuracy of scientific contents, this disclosure certifies
that in the past 2 years:
“I have had no collaboration with companies with
commercial interests operating in the healthcare area,
and I have not been involved in consulting
assignments.”

3. Alternative, but adequate study designs for rare
diseases that can convince coverage-decision makers;
What can we learn from Sun Tzu (Chinese military strategist)?
Thomas Hoogeboom, PhD
Senior Researcher at IQ healthcare
Radboud Institute for Health Sciences
Radboud university medical center

4. x
Bladder instillations with
glycosaminoglycans (GAG)
Bladder pain syndrome with
hunner lesions
Lack of evidence for
its effectiveness
New high quality
evidence on
effectiveness
But how do we know whether
a treatment is effective?

5. If you know the enemy and
know yourself you need not
fear the results of a hundred
battles.

6. If you know the enemy and
know yourself you need not
fear the results of a hundred
battles.

7. If you know the coverage-
decision maker and know
yourself you need not fear the
results of a hundred grant
applications.
(a loose
interpretation)

8. Effectiveness is established when:
The (new) intervention contributes to the
objective intended by the intervention in
comparison with what is already offered in
practice for care for the disorder in question.
Note: Effectiveness relates to the achievement
of a clinically relevant goal(s) in daily practice.
Disclaimer: This report does not reflect my own views on effectiveness
Good read: Huang and Hood, Perspectives in Biology and Medicine (2019)

9. RCTs
Observational
studies
BPS with hunner lesions
accounts for ~600 clinical
patients in the NL.!
Patients rather not
receive a placebo.!
https://www.gradeworkinggroup.org/

10. But they won’t really allow evidence from
other sources than the RCT right?

11. Well, actually they do..
11 POSITIVE
reimbursement
decision
11 NEGATIVE
reimbursement
decision
de Groot et al, BMJ Open (2015)

12. 11 POSITIVE
reimbursement
decision
11 NEGATIVE
reimbursement
decision
Evidence from (blinded)
RCTs lacking
de Groot et al, BMJ Open (2015)
Well, actually they do..

13. 11 POSITIVE
reimbursement
decision
11 NEGATIVE
reimbursement
decision
Evidence from (blinded)
RCTs lacking
Evidence from (blinded)
RCTs lacking
de Groot et al, BMJ Open (2015)
Well, actually they do..

14. So, when dealing with small samples, what study
design is useful?
(Important design ingredients: randomization, blinding, and control group)
Heymans et al, NTvG (2013)

15. Combined N-of-1 trials
The supreme art of war is to subdue the enemy without fighting.
- Sun Tzu

16. Guyatt, et al. JAMA (2002); Vohra et al, BMJ (2015); Schork, Nature (2015);
Oxford Centre for EBM Levels of Evidence Working Group (2011).

17. http://www.consort-statement.org/extensions/overview/n-of-1

18. Group A
(n=40)
Baseline
Placebo
(6 wk)
Wash-out
(8 wk)
Intervention
(6 wk)
Wash-out
(8 wk)
Intervention
(6 wk)
Run-out
(2wk)
Group B1
(n=20)
Baseline
Intervention
(6 wk)
Wash-out
(8 wk)
Placebo
(6 wk)
Wash-out (8
wk)
Intervention
(6 wk)
Run-out
(2 wk)
Group B2
(n=20)
Baseline
Intervention
(6 wk)
Wash-out
(8 wk)
Intervention
(6 wk)
Wash-out
(8 wk)
Placebo
(6 wk)
Run-out
(2 wk)
So what did we propose?
Combined n-of-1 study (n=28)
Randomized clinical trial
Randomized? √
Blinded? √
Control group? √
The supreme art of war is to subdue the enemy without fighting.
- Sun Tzu

19. Group A
(n=40)
Baseline
Placebo
(6 wk)
Wash-out
(8 wk)
Intervention
(6 wk)
Wash-out
(8 wk)
Intervention
(6 wk)
Run-out
(2wk)
Group B1
(n=20)
Baseline
Intervention
(6 wk)
Wash-out
(8 wk)
Placebo
(6 wk)
Wash-out (8
wk)
Intervention
(6 wk)
Run-out
(2 wk)
Group B2
(n=20)
Baseline
Intervention
(6 wk)
Wash-out
(8 wk)
Intervention
(6 wk)
Wash-out
(8 wk)
Placebo
(6 wk)
Run-out
(2 wk)
So what did we propose?
Combined n-of-1 study (n=28)
Randomized clinical trial
Randomized? √
Blinded? √
Control group? √
The supreme art of war is to subdue the enemy without fighting.
- Sun Tzu

20. 8wk 8wk
8wk 8wk
RCT
Combined N-of-1 trials

21. Proposal was accepted
To our knowledge first in its kind

22. I wondered… could I come up with two alternative
designs for this BPS challenge for this talk?
Never venture, never win!
― Sun Tzu

23. “De Novo” Prognostic Meta-analysis
Opportunities multiply as they are seized.
- Sun Tzu

24. Seidler et al, BMJ (2019)

25. De Novo Prognostic (or pre-planned) meta-analysis
• Multiple (under-powered) RCTs
• At different sites/countries:
• Interventions protocols adapted to context
• Statistical analyses and variations in
methods/procedures
• Statistically pooled with individual patient data
meta-analysis
In ‘t Hout et al, Stat Meth Med Res, 2016

26. Seidler et al, BMJ (2019)

27. Fictional study:
• RQ—Does GAG therapy reduce pain in people with BPS and hunner lesions?
• Design—Preplanned meta-analysis of 5 independent RCTs.
• Setting—5 academic hospital and 2 private clinics.
• Participants—15 to 25 per study (statistical power ~30%).
• Interventions—GAG therapy administration can vary across studies in
duration, dosage, frequency, and brand. Perhaps some sites combine GAG
therapy with behavioral components.
Prognostic meta-analysis
Randomized? √
Blinded? √
Control group? √
Embrace heterogeneity!

28. Province et al, JAMA, 1995

29. Headline:
First “De Novo” Prognostic Meta-analysis in the field of Urology
started at ESSIC 2019.
Opportunities multiply as they are seized.
- Sun Tzu

30. Regression Discontinuity Design
(on retrospective data from real-world practice)
All warfare is based on deception
- Sun Tzu

31. Simple Design
Pre-intervention Intervention Post-intervention
Above cut-off O X O
Below cut-off O O
For example:
GAG treatment with a bladder instillation for people with BPS with Hunner
lesions, can be continued after 6 washes if there is clinical improvement
on VAS pain of 2 points.

32. Pre-Post Null Case
1009080706050403020100
80
70
60
50
40
30
20
10

33. Pre-Post Null Case
1009080706050403020100
80
70
60
50
40
30
20
10
Pre
Postnull
Regression
line

34. RD Design, No Effect
1009080706050403020100
80
70
60
50
40
30
20
10
Pre
Postnull
Cutoff

35. RD Design with Effect
1009080706050403020100
80
70
60
50
40
30
20
Pre
Posteff
Cutoff

36. RD Design with Effect
1009080706050403020100
80
70
60
50
40
30
20
Pre
Posteff
Cutoff
Treatment group
Control group

37. RD Design with Effect
1009080706050403020100
80
70
60
50
40
30
20
Pre
Posteff
Cutoff
Treatment group
Control group
Difference between
the regression lines
at the cut-off value is
the treatment effect.
Randomized: X
Blinded: X
Control group:√
Why does this design work??
Because individuals around cut-off are
very similar on (un)observed
characteristics (as if randomly
assigned).
All warfare is based on deception
- Sun Tzu

38. Several advantages RD Designs:
• Treatment effect estimates similar to that of RCTs.
• No randomization thus no ethical issues of random assignment.
• Real world populations.
• Applicable to neglected sub-groups.
Rubin, Journal of Educational and Behavioural Statistics (1977);
Moss et al, Educational Evaluation and Policy Analysis (2014);
O’Keeffe et al, BMJ (2014).

39. All in all…
Choosing the right study design is pretty important..

41. Dick Janssen
Thank you for your attention!
It is even more important to: