Mixed Methods Research incorporating Design Development framework.

1. Mixed Method Educational Research
Mental Simulation
Case Example
Vaikunthan Rajaratnam 1

2. Formulate Research Questions
In simulation research
https://tinyurl.com/S3Research22

3. Problem statement
• Describe issue that needs a solution
• Provides purpose and intent of research
• Brief and Concise
• What is known?
• What is not known
• Allows for focus to explore the problem
• Its impact and significance
• So what!
CONTEXT
• Problem/
Solution
Gap
SIGNIFICANCE
• Ideal state
SOLUTION
• Reaching
the ideal

4. Template for problem
statement
This research addresses the problem
of (insert significant issue/problem
with the educational program)… if left
unsolved would create ( describe the
undesirable state of the educational
process/activity)…can be resolved by
(describe your research design)…
producing …(describe the ideal state of
the educations program)

5. Simulation-based HPE research
Curriculum
Community
orientated
Performance
gap
Simulation
deliverable
Pedagogy
Simulation
deliverable
Innovate
(AR/VR/AI)
Affordable
and
accessible
Assessment
Formative
Portfolio
Summative

6. FINER framework
Feasible
• Answerable
• Potential
participants
• Enough time
• Attainable
objectives
• Effect size
• Resources and
expertise
• statistician or
methodologist
• conducted a pilot
•Interesting
• Rewarding
• Useful
• Application
• Worthy
• Needed
• Multidisciplinary
Novel
• Search for prior
questions
• New in which
domain
• Improve rigour
from previous
• Bridging gap in
knowledge
Ethical
• Risk
• Benefits
• Safety
• Consent
• Incentives
• Data Protection
• Dissemination of
findings
Relevant
• Improve
Outcome
• Meet needs
• Quality
• Value
• Translatable
• Implication on
current practice

7. PICO framework
POPULATION
• Problem
• Inclusion
• Exclusion
• Demographic
• Specific characteristics
•INTERVENTION
• Predictor/Independent
variable
• Outcome/Dependent
variable
• Theoretical
or Conceptual
framework
• Pre/post and
multivariate outcomes
COMPARATOR/CONTEXT
• Control group
• Gold standard
• Magnitude, variance,
and direction of effect
• Baseline or preliminary
measures a control
• Context
OUTCOMES
• Dependent variable
• Explicitly stated
• Highest possible scale
• Credible and Valid
• Continuous outcome (?)
• Creating research
hypotheses
• Sample size
• Scale of measurement
• Choice of analysis

8. Identify
Needs
Task and
competence
level
Clear
learning
objectives
Demonstrate/
Introduce
task
Practice
Assess
Performance
Measure
outcome
Feedback
and review
Simulation-Based Training

9. Problem Statement
PICO framework
Population
• Learners in
Singapore
Comparator
• Undergoing
the
stipulated
period of
training
Intervention
• Module
with motor
imagery
and mental
practice
(MIMP)
Outcome
• Predictable
level of
expertise
Vaikunthan Rajaratnam 9

10. Purpose of the Study
Design and develop an online instructional program
Incorporates MIMP
Enable novice learners
Acquire motor skill of micro suturing.
Level of mastery
Vaikunthan Rajaratnam 10

11. Research Objectives
Design and
develop an
online training
module that
incorporates
MIMP
Validate and
evaluate the
module
Evaluate the
module's
usability level
for novice
medical student
Describe a
model and tool
for developing
skills training
materials
Vaikunthan Rajaratnam 11

12. Research Questions
1.How can an instructional module be designed
and developed for deliberate practice in
microsurgery, incorporating Motor Imagery &
Mental Practice (MIMP) to produce expert
performance among novices?
2.Will this module produce expert measurable
skill level in a standardised micro suturing
procedure for medical students, as compared to
the current methods of training in Singapore?
3.How does the module allow for ease of use for
learning basic microsurgery?
4.How can the process of this research be used
as a model and tool for developing instructional
materials for surgical skill acquisition using
MIMP?
Vaikunthan Rajaratnam 12

13. Research Hypotheses
Null hypothesis (H0)
• Incorporating MIMP into a microsurgery suturing training module (the intervention)
does not improve the rate and level of acquiring competencies in microsurgical
suturing (the outcome) in a rubber glove model among medical students (population)
as compared to the available online resource and face to face demonstration (the
comparator) in Singapore.
Alternate hypothesis (H1):
• MIMP incorporated into a microsurgical suturing training module significantly
improves the rate and level of acquiring competencies in microsurgical suturing in a
rubber glove model among medical students as compared to the available online
resource and face to face demonstration in Singapore.
Vaikunthan Rajaratnam 13

14. Vaikunthan Rajaratnam 14
Rajaratnam, V., Rahman, N. A., & Dong, C. (2021). Integrating instructional design principles into surgical skills training models: An innovative
approach. The Annals of The Royal College of Surgeons of England, 103(10), 718–724.

15. Ideomotor Theory
• To trigger an action, the effects of the action are anticipated
(effect anticipation).
• Anticipatory image of action-effects directly activates an action
by means of direct associations between actions and
effects (direct activation).
• During learning these associations are acquired
independently of the actor’s current intentions and possibly
without the help of a teacher (associative learning rule).
Vaikunthan Rajaratnam 15

16. Covert
stage
Representation
of the future
A goal
The means
The
consequences
Overt
stages
Physically
executed
Covert – Overt
continuum
Overt must
have covert
Covert need
not lead to
overt
Motor simulation theory
16

17. Theoretical basis
Vaikunthan Rajaratnam 17

18. Conceptual Framework
Micro suturing skill - High Acuity Low Opportunity activity
Vaikunthan Rajaratnam 18

19. Vaikunthan Rajaratnam 19

20. Design Key Features
Convergent Parallel Simultaneous independent data collection
and analysis. Results merged to provide a
more complete understanding of a
phenomenon.
Explanatory
Sequential (Quantitative first)
Strands occur in turn, with initial
quantitative results being explained in
more detail through qualitative enquiry.
Exploratory
sequential (Qualitative first)
Methods carried out sequentially, with the
qualitative phase first. Quantitative method
then seeks to test and further quantify
initial qualitative findings.
Embedded An additional strand of research is added to
a larger study to gather supplemental
information about this.
Vaikunthan Rajaratnam 20
MIXED METHOD DESIGN

21. Design and Development Research
(Richey & Klein, 2007)
Product and Tool Research Model Research
Emphasis Study of specific product or
tool design and development
projects
Study of model
development, validation,
or use
Outcomes Lessons learned from
developing specific products
and analysing the conditions
which facilitate their use
New design and
development procedures
or models, and conditions
which facilitate their use.
Context-specific Generalised
Vaikunthan Rajaratnam 21

22. Research Design Concern Recommended techniques used to address
concern
How Study Addressed Concern
Validity • Use experts with areas of specialization for
model and module review.
• Have participants verify reports of module
use.
• Select doctors with varying levels of
expertise to validate the module.
• Five expert microsurgeons/ reviewers with broad
knowledge of performance in microsurgery
• Participants involved in the test of the tool
confirmed their status as practicing microsurgery
• Participants were selected from different grades
to assess the modules that will be developed.
Causal Inferences • Determine module’s practicality
• Determine module’s effectiveness.
• The module was made easily accessible via online
access for the participants to review
• Survey questions were deployed to ensure face
and content validity.
Generalization and
Interpretation
• Recognize “real world” constraints on
module’s use.
• Plan for module’s independent use.
• Non-Stakeholders were used to review and assess
the module using online survey tool.
• Made module accessible as an open educational
resource for others to use.
Anticipating
problems
• Consider non-laboratory and different
cultural contexts when analysing student
data.
• In analyses, consideration was given that this is a
small sample experimental study, with its
limitation.
Vaikunthan Rajaratnam 22

23. Vaikunthan Rajaratnam 23
Design and Development
Research Framework
MODEL
PRODUCT & TOOL

24. Proposed Model
Vaikunthan Rajaratnam 24

25. Objectives Question/Hypothesis Methodology Outcomes/Variables Analysis Conclusions
Design, develop and
validate a training
module in micro
suturing
How can an instructional module
be designed and developed for
deliberate practice in
microsurgery, incorporating
Motor Imagery & Mental Practice
(MIMP) to produce expert
performance among novices?
Expert review, in-depth
interview,
CWT process and
validation
Task and subtask
analysis, mentals
script development,
content
development and
incorporation into
learning module
VPA/HTA for task
analysis.
MIMP script
developed and
incorporated into
instructional video
and module.
To evaluate the
effectiveness of this
module against the
current training
module
Will this module produce expert
measurable skill level in a
standardised micro suturing
procedure for medical students,
as compared to the current
materials for micro suturing
training in Singapore?
Experimental study,
survey
Performance - time
taken to complete
task SMART
instrument.
Quality of motor
imagery measured
with MIQ, score.
Non-parametric
testing Mann-
Whitney test
Nonparametric
Spearman rho
correlations
Cronbach Alpha
Endorse module with
MIMP
To evaluate the ease
of
learning of this
module
Does the module allow for ease of
learning basic microsurgery?
Survey and in-depth
interview
Cognitive load
NASA TLX index
Descriptive analysis Recommend MIMP
incorporation in the
programme
To evaluate the
usability of learning of
the new module
Survey and in-depth
interview
Usability attributes
SUS score
Descriptive analysis Recommend use of
MIMP in module
Vaikunthan Rajaratnam 25
Research Design Matrix

26. Research Procedures
Vaikunthan Rajaratnam 26

27. Vaikunthan Rajaratnam 27

28. Vaikunthan Rajaratnam 28

29. Vaikunthan Rajaratnam 29

30. Vaikunthan Rajaratnam 30
Codes
Number of
coding
references
Aggregate
number of
coding
references
Number
of items
coded
Aggregate
number of
items coded
CodesAssess Outcome 55 119 5 7
CodesAssess
OutcomeFinal
Inspection
14 14 5 5
CodesAssess
OutcomeKinaesthetic
and spatial awareness
50 50 6 6
CodesDrive Needle 39 102 6 7
CodesDrive
NeedleForceps
manipulation for
grasping
11 11 5 5
CodesDrive
NeedleNeedle Grasping
17 17 5 5
CodesDrive
NeedleNeedle
movement and tissue
engagement
18 18 6 6
CodesDrive
NeedleNeedle Use
10 10 4 4
CodesDrive
NeedleSuture grasping
7 7 5 5
CodesPreparation 34 105 5 9
CodesPreparationInstr
ument handling
20 40 6 8
(Table 4.8 continued)
CodesPreparationInstr
ument handlingForceps
14 14 7 7
CodesPreparationInstr
ument handlingNeedle
Holder
5 5 4 4
CodesPreparationInstr
ument handlingScissors
1 1 1 1
CodesPreparationMicr
oscope
1 5 1 2

31. Vaikunthan Rajaratnam 31
Hierarchical Chart of the frequency of the transcript references to all the codes.
Reproduced from the imported data of the research from QSR International Pty Ltd. (2020) NVivo

32. Vaikunthan Rajaratnam 32
Hierarchical Task analysis form the thematic analysis of the transcripts

33. Expert
Task/Subtask
A B C D E
1 : Assess Outcome 8% 37% 20% 17% 17%
2 : Final Inspection 25% 43% 19% 0% 13%
3 : Kinaesthetic and spatial awareness 9% 35% 20% 19% 18%
4 : Drive Needle 16% 40% 17% 15% 11%
5 : Forceps manipulation for grasping 4% 41% 28% 17% 10%
6 : Needle Grasping 21% 22% 25% 23% 9%
7 : Needle movement and tissue engagement 18% 44% 15% 14% 9%
8 : Needle Use 52% 14% 18% 17% 0%
9 : Suture grasping 10% 38% 6% 38% 8%
10 : Preparation 6% 45% 24% 19% 7%
11 : Instrument handling 6% 46% 28% 13% 7%
12 : Forceps 16% 33% 30% 12% 8%
13 : Needle Holder 0% 18% 26% 47% 8%
14 : Scissors 0% 100% 0% 0% 0%
15 : Microscope 0% 100% 0% 0% 0%
16 : Adjusting eyepiece 0% 100% 0% 0% 0%
17 : Magnification 88% 12% 0% 0% 0%
18 : Positioning of body and upper limb 0% 56% 0% 44% 0%
19 : Hand 0% 56% 0% 44% 0%
20 : Legs. 0% 0% 0% 0% 0%
21 : Neck 0% 0% 0% 0% 0%
22 : Sit 0% 0% 0% 100% 0%
23 : Positioning of field 0% 74% 0% 26% 0%
24 : Positioning of rubber glove model 12% 37% 18% 23% 9%
25 : Rubber Glove Model
Table 4.11 continued
0% 34% 43% 23% 0%
26 : Suture Cutting 25% 43% 13% 5% 13%
27 : Length assessment 24% 40% 0% 0% 35%
28 : Orientation 54% 36% 0% 0% 10%
29 : Scissors manipulation 24% 44% 0% 17% 14%
30 : Suture tying 19% 37% 16% 14% 14%
31 : Create Knot 12% 41% 15% 20% 13%
32 : Secure knot 32% 24% 23% 8% 12%
33 : Withdraw Suture 11% 43% 13% 19% 15%
34 : Grasp Suture 0% 0% 100% 0% 0%
35 : Pulling of suture 7% 43% 12% 24% 15%
Vaikunthan Rajaratnam 33

34. Vaikunthan Rajaratnam 34

35. Name Codes References
A 28 1854
B 36 3775
C 27 2202
D 30 2272
E 27 1386
Vaikunthan Rajaratnam 35
Assigned references and codes to experts transcripts

36. Vaikunthan Rajaratnam 36

37. Vaikunthan Rajaratnam 37

38. Dissemination
1. Rajaratnam, V., Ng, H. J. H., Rahman, N. A., & Dong, C. (2022a). Development of a mental script for the
mental practice of micro suturing: A methodological approach. The Annals of The Royal College of Surgeons
of England, 104(4), 288–294.
2. Rajaratnam, V., Ng, H. J. H., Rahman, N. A., & Dong, C. (2022). Online training module for micro suturing
incorporating motor imagery and mental practice: A design and development research study. ANZ Journal
of Surgery, 92(9), 2072–2081. https://doi.org/10.1111/ans.17772.
3. Rajaratnam, V., Rahman, N. A., & Dong, C. (2021). Integrating instructional design principles into surgical
skills training models: An innovative approach. The Annals of The Royal College of Surgeons of England,
103(10), 718–724.
4. Rajaratnam, V., Rahman, N. A., Dong, C., & Ng, H. J. H. (2022). Redefining Surgical Skill Acquisition. Medical
Education for the 21st Century, 121.
5. Rajaratnam, V., Rahman, N. A., Omar, U. F., & Ng, H. J. H. (2021). Mental practice in surgical training. The
Bulletin of the Royal College of Surgeons of England, 103(8), 403–408.
Vaikunthan Rajaratnam 38