Clinical simulation evaluates health IT safety

Clinical Simulation as an Evaluation Method
in Health Informatics
Sanne JENSEN
a,1
a
The Capital Region of Denmark, Copenhagen, Denmark
Abstract. Safe work processes and information systems are vital
in health care.
Methods for design of health IT focusing on patient safety are
one of many
initiatives trying to prevent adverse events. Possible patient
safety hazards need to
be investigated before health IT is integrated with local clinical
work practice
including other technology and organizational structure.
Clinical simulation is
ideal for proactive evaluation of new technology for clinical
work practice.
Clinical simulations involve real end-users as they simulate the
use of technology

in realistic environments performing realistic tasks. Clinical
simulation study
assesses effects on clinical workflow and enables identification
and evaluation of
patient safety hazards before implementation at a hospital.
Clinical simulation also
offers an opportunity to create a space in which healthcare
professionals working
in different locations or sectors can meet and exchange
knowledge about work
practices and requirement needs. This contribution will discuss
benefits and
challenges of using clinical simulation, and will describe how
clinical simulation
fits into classical usability studies, how patient safety may
benefit by use of
clinical simulation, and it will describe the different steps of
how to conduct
clinical simulation. Furthermore a case study is presented.
Keywords. Ergonomics, eHealth, qualitative evaluation, clinical
simulation, risk,
safety.
1. Introduction
Implementation of health IT in relation to improvement of

patient safety and
optimization of work flow is a paradox [1]. Even though health
IT is intended and
anticipated to have a positive impact on quality and efficiency
of health care [2], the
application of new technology in healthcare may also increase
patient safety hazards [3,
4]. Studies show that adverse events are indeed often related to
the use of technology
[5-7].
Design of health IT focusing on protecting patient safety is one
of many initiatives
trying to prevent adverse events [8, 9].
2
Patient safety does not entirely rely on
technology but is highly influenced by the interaction between
users and technology in
a specific context [10], and sociotechnical issues and human
factors are related to many
unintended consequences and patient safety hazards [7, 8, 11].
Possible patient safety
hazards such as design of the IT system itself; embedding of IT
system into local work

1
Corresponding author: Sanne Jensen, The Capital Region of
Denmark, Borgervanget 7, 2100
Copenhagen O, Denmark, [email protected]
2
See also: F. Magrabi et al., Health IT for patient safety and
improving the safety of health IT, in: E.
Ammenwerth, M. Rigby (eds.), Evidence-Based Health
Informatics, Stud Health Technol Inform 222, IOS
Press, Amsterdam, 2016.
Evidence-Based Health Informatics
E. Ammenwerth and M. Rigby (Eds.)
© 2016 The authors and IOS Press.
This article is published online with Open Access by IOS Press
and distributed under the terms
of the Creative Commons Attribution Non-Commercial License.
doi:10.3233/978-1-61499-635-4-152
152
practice in the local environment; and the introductory and
training processes need to
be investigated when health IT is integrated with local clinical
work practice including
technology and organizational structures. The substantial

complexity of organizations,
work practices and physical environments within the healthcare
sector impacts design,
evaluation and implementation of information systems [12, 13].
Healthcare
environments are profoundly collaborative and rely on
coordination between various
health professionals [14]. They are characterized by delegated
decision-making,
multiple viewpoints and inconsistent and evolving knowledge
bases [15]. Multiple
groups with potentially divergent values and objectives work
together and face many
contingencies which cannot be fully anticipated [16, 17]. These
matters challenge
design and evaluation of health IT.
Clinical simulation tries to address this challenge. Compared
with other methods,
e.g. heuristic inspection and low fidelity usability evaluation
3
, clinical simulation takes
the clinical context into account. In contrast, for example,
heuristic inspection focuses
on the user interface, and low fidelity usability testing focuses

on technology and on
the specific tasks of individual users. By including the clinical
context, clinical
simulation is ideal for proactive evaluation of new technology
for clinical work practice
[18, 19]. Clinical simulations involve real end-users as they
simulate the use of
technology in realistic environments performing realistic tasks
[20]. Clinical simulation
studies the effects on clinical workflow [21] and enables
identification and evaluation
of patient safety hazards before implementation at a hospital or
other clinical setting
[22]. Clinical simulation also offers an opportunity to create a
space in which
healthcare professionals working in different locations or
sectors can meet and
exchange knowledge about work practices and requirement
needs [23, 24].
Hospital organizations and work practice are extremely complex
with many
different healthcare groups and many interactions and
correlations [25] involved, and
many acute situations are encountered during daily work

practice in hospital settings
[26]. This complexity affects the technology that is
implemented at hospitals [27] and
confronts the methodology used for design and evaluation of
healthcare information
systems. Failure to comprehend the nature and range of end-
users has been highlighted
as a key issue in many systems’ failing to become accepted by
healthcare professionals
[28]. Furthermore, an understanding of the context in which the
systems will be used
must take into account not only tasks and settings [29], but also
the range, competences
and cognitive capacities of an increasing variety of potential
end-users [30]. The risk of
endangering patient security calls for careful evaluation before
implementing new
technology in real life settings [31].
Usability relates to how a product can be used to achieve
specified goals with
effectiveness, efficiency and satisfaction in a specified context
of use.
3
Usability

focuses on use of technology in a specific context [32]. Context
may be defined as
“users, tasks, equipment (hardware, software and materials),
and the physical and
social environment in which a product is used” [32]. It does
however raise several
questions, e.g. ‘who are the users?’, ‘what are their tasks?’ and
‘with whom, where and
under what conditions are they performing these task?’. The
healthcare sector poses
challenges due to the larger potential numbers and classes of
users, e.g. nurses,
physicians and pharmacists [28]. Furthermore, the definition
does not take multiple
3
See also: R. Marcilly et al., From usability engineering to
evidence-based usability in health IT, in: E.
S. Jensen / Clinical Simulation as an Evaluation Method in
Health Informatics 153

users and their professional interaction into account, and nor
does it take parts of or a
whole organization into account.
According to Hertzum [33] many views may be put on usability,
dividing it into
six images; 1) universal usability: usability in a system for
everybody to use, 2)
situational usability: quality-in-use of a system in a specified
situation with its users,
tasks, and wider context of use, 3) perceived usability: usability
concerns the user’s
subjective experience of a system based on her or his
interaction with it, 4) hedonic
usability: usability is about joy of use rather than ease of use,
task accomplishment, and
freedom of discomfort, 5) organizational usability: usability
implies groups of people
collaborating in an organizational setting, and 6) cultural
usability: usability takes on
different meaning depending on the users different background.
Hertzum claims that all
images should be taken into account when evaluating usability.
Another aspect when designing and evaluating information
systems is user

involvement. User-centred design focuses on incorporating the
user’s perspective into
the development process in order to attain a usable IT system
[34].
4
The key principles
of user-centred design are 1) active involvement of users and
clear understanding of
user and task requirements; 2) an appropriate allocation of
function between user and
system; 3) iteration of design solutions; and 4) multi-
disciplinary design teams. The
human-centred design cycle [32] shown in Figure 1 describes
five essential processes
which should be undertaken in order to incorporate usability
requirements into the
software development process.
Figure 1: The human-centred design cycle.
The process is iterative with the cycle being repeated until the
particular usability
objectives have been obtained. Studies show that effective
involvement of users may
leads to 1) improved quality of the system arising from more
accurate user

requirements; 2) avoidance of costly system features that users
do not want or cannot
4
See also: A. Kushniruk et al., Participatory design and health
IT evaluation, in: E. Ammenwerth, M.
Rigby (eds.), Evidence-Based Health Informatics, Stud Health
Technol Inform 222, IOS Press, Amsterdam,
2016.
Health Informatics154
use; 3) improved levels of acceptance of the system; 4) greater
understanding of the
system by the user resulting in more effective use; and 5)
increased participation in
decision-making in the organization [35, 36].
2. Clinical simulation
Clinical simulation supports involvement of context as well as
end-users in pre-
implementation design and evaluation of health IT. Clinical
simulations involve real
end-users as they simulate the use of technology in realistic

environments performing
realistic tasks [20]. As shown in Figure 2, clinical simulation
can be used in different
evaluation activities at various phases of the development life
cycle from evaluation of
work practice and user requirements, evaluation of the initial
specification and early
design solution so as to seek to eliminate patient risks created
or perpetuated, through
to application assessment in work practice and assessment of
training programs.
Patient safety issues may be explored in all phases of the
lifecycle by observing
and analysing medical errors and work flow in a simulated
situation close to a real life
environment [22]. In the first phases of the lifecycle of health
IT, simulation may be
used for specification and evaluation of user requirements [19],
as well as for obtaining
knowledge and evaluate work practice [37]. This may involves
observation of
clinicians applying information technology under simulated
conditions
Figure 2. Simulation evaluations in information system life

cycle.
Likewise in the design phase simulation is well suited as a
method for user
involvement in connection with evaluation of the design.
Simulation studies can be
designed to gain practical experience in evaluation of new
technology without
introducing any kind of ethical issues and without putting
patients at risk [20]. In this
way it is possible to test prototypical software in realistic
scenarios and environments.
Simulations can be performed in laboratories as well as in situ
in a ward, an operating
theatre or an outpatient clinic [20]. Simulation studies aim to
evaluate design proposals
for a new technology and combine elements of the laboratory
test and the field study
[22].
Particular aspects of implementation can be visualized by
simulation e.g. user
interaction in work practice, the need for training, and the
impact of decision support
[22]. Unintended consequences of new systems such as changes
in work processes and

patient outcome
5
may also be detected and can provide constructive and valuable
5
See also: F. Magrabi et al., Health IT for patient safety and
improving the safety of health IT, in: E.
information for organizational decision makers [18]. Clinical
simulation can also be
used as common ground for discussion and negotiation and as
an organizational
learnings space, where knowledge of other parts of an
organization can be acquired
[37].
The realism and acceptance of the simulation depend on the
degree of fidelity in
the simulation set-up. Dahl and colleagues [38] have developed
a simulation

acceptance model
with four fidelity dimensions: 1) environment – physical
elements,
such as rooms, beds and patient; 2) equipment – elements, such
as mock-ups and
electronic devices; 3) functionality – such as system
functionalities and interactive
devices; and 4) tasks – clinical task such as administration of
drugs and ward rounds.
These fidelity dimensions affect the perceived realism and
thereby acceptance of the
simulation made by the involved clinicians and should be
considered carefully
according to the purpose of the simulation.
Clinical simulations are performed in three phases; 1)
introduction, 2) simulation,
and 3) evaluation. Prior to the simulation, the participants are
introduced to the
information system and to the simulation. Simulation facilities
are a dedicated facility
with two rooms linked by a one-way mirror. During the
simulation, a simulation
facilitator is located in the simulation room. The facilitator

assists the simulation and
supports the participating clinician. An instructor located in the
observation room
instructs the patient and the simulation facilitator. A one-way
mirror separates the two
rooms. The simulation is observed by health informatics experts
and sometimes by key
stakeholders, such as colleagues from hospitals, clinical
managers, quality managers
and vendors [37]. The observers are located in the observation
room.
An example of how simulation facilities may look like is
presented in Figure 3.
The simulation room is established as a bed room for two
patients with bedside tables
and a portable for the healthcare professional. An observation
room with portables and
chairs is located in the right corner. A one-way mirror is
separating the two rooms.
Figure 3. Overview of the physical simulation set-up.
Simulation of handover from hospital to community care by
messaging

technologies can also be carried out in a simulation laboratory.
In such situations
another simulation room may replicate a nursing office at the
community care. In
situations where it is not possible to replicate the location of the
simulation in a
laboratory, simulation in situ may be used. This could be
scenarios where large x-ray
scanners or other large equipment is involved.
The simulating clinicians are asked to “think aloud” so that the
observers can
acquire a deeper understanding of the human task-behaviour.
Depending on the
purpose of the clinical simulation, the clinicians are sometimes
also able to observe
their colleagues, when not participating in the simulation
themselves [39]. The different
roles and their locations are described in Table 1.
Table 1. Overview and description of different roles and their
locations during clinical simulation.

Roles Description Location
Instructor Overall responsible for the simulation. Instructs
simulation facilitator and patient(s) during simulation by
use of intercom equipment and facilitates debriefing.
Observation room
Simulation
facilitator
Briefs clinicians prior to simulation and provides support
during simulation. Receives instructions from and assists
instructor during simulation.
Simulation room
Observers Observes and makes notes during simulation; e.g.
usability, support of work practice, patient safety
Observation room
Clinicians Simulates scenario. Thinks aloud during simulation.
Participates as interviewee in interview
Simulation room
Actor Acts as e.g. patient, colleague during simulation and

receives instructions from instructor.
Simulation room
After the simulation, the proposed information system is
evaluated. Participants
are asked to complete questionnaires and participate in a de-
briefing interview.
Additional to interview guides, observations made by the
observers during the
simulations are used as background for the interviews [24]. It
must be clarified in
advance to whom the results are to be presented and how the
results and
recommendations should be implemented. The same goes for the
respective mandates
of the participating clinicians as well as the observers.
3. Case Study: Simulation study of a clinical information
system
The aim of the case study was to investigate how a newly-
acquired standard clinical
information system for doctors to sign for laboratory results
might support clinical
practice, and to identify potential patient safety hazards prior to
its implementation [40].
The aim of the information system was to obtain an IT

supported work flow for
physicians receiving and signing laboratory test results in order
to improve patient
safety. In addition to implementation aspects such as training
and information, the
purpose was also to evaluate future work practice, the relation
between technology and
existing work processes, and the extent to which clinical
simulation may be applied as
a proactive method to identify and evaluate potential patient
safety hazards prior to
implementation.
The existing workflow was paper based; i.e. prints were made
from digital systems
and were signed by a doctor in order to document that the
specific test result had been
reviewed by a doctor. The laboratory tests were handled by
various information
systems. Some test results were on paper and others were
electronic. The background
for the local work flows was based on interpretations of a
national guideline for
handling laboratory test results. This national guideline was
developed as part of a

quality assurance initiative to increase patient safety. As a rule,
the physicians signed to
confirm that they have seen a laboratory test result. The
physicians also signed to
confirm that they have handled the test results. The essential
challenges about the paper
based workflow were 1) lack of overview about whether a result
has arrived; 2)
uncertainty about whether a test result has been seen by a
physician; and 3) lack of
documentation about which physician has seen a test result.
The objective of purchasing the new information system was to
increase quality in
work practice and minimize the risk to patient safety by
implementing a new standard
information system. The information system collects laboratory
test results and
supports electronically documentation of acknowledging the
results. The study was
expected to be moderate and manageable because the

information system was a
standard off-the-shelf product and the intended work flow was
supposed to be narrow
and well-defined. The information system was to be
implemented at two pilot
departments. Both departments included patient wards and
outpatient clinics. Prior to
implementation, the existing work practice was analysed and
future generic work flows
defined. The functionality of the information system and
collaborative future work
practice were evaluated by means of clinical simulation. The
aim of the simulation
study was to assess how the information system supported
clinical practice and to
identify potential patient safety hazards prior to its
implementation.
Initial field studies were carried out at the two pilot
departments covering both
patient wards and outpatient clinics in order to gain insight into
existing work practice
concerning receipt, handover and acknowledgement of
laboratory test results. Two
workshops were then held with physicians, nurses and medical

secretaries from the
pilot departments, health informaticians and experts from the
regional quality unit. At
the first workshop, future work practice and the information
system were analysed and
required changes were identified. At the second workshop,
future work practice was
determined, focusing on improved efficiency, quality,
continuity and communication.
Existing routines were contested and organizational changes
were initiated ahead of
implementation to create acceptance and a readiness to change
among future end-users.
An analysis of work practice conducted prior to the clinical
simulation revealed
that there were significant differences between the hospitals,
between the patient wards,
and the outpatient clinics – and indeed also between the
individual healthcare
professionals. Furthermore, the design of future work practice
presented a number of
challenges and it was not possible to design a generic work flow
to cover both patient
ward and outpatient clinic. This was to some extent due to

differences between local
work flows but also due to the fact that the information system
functionality did not
provide adequate support for work practice.
Clinical simulation was conducted after the two workshops. The
purpose of the
clinical simulation was to evaluate patient safety issues and
future work practice using
the new information system before its implementation. Six
healthcare professionals
from the two pilot departments (two physicians, three nurses
and one medical
secretary) participated in the simulations. Clinical managers
from the pilot sites,
implementation experts and health informatics experts were
observing the simulations.
Figure 4 shows the simulation room seen from the observation
room through a one way
mirror. The simulation set-up is an outpatient clinic where a
physician is preparing for
a meeting with a patient.
A total of 11 scenarios were performed during the simulation;
six scenarios from

patient wards and five scenarios from outpatient clinics. All
scenarios were related to
signing and handling laboratory test results. Some of these were
frequently performed
work flows, e.g. ward rounds and visits to the outpatient clinic,
while others were
critical work flows; e.g. urgent test results, sorting test results
and handover of
responsibility. The simulation set-up was very realistic. The
computers used were
identical with those used at the hospitals and the system was
fully developed and
operational. The scenarios were composed in participation with
clinicians from the
pilot sites and based on realistic patient cases. The simulation
room was designed as
either a ward bedroom or clinical office. The role of patient was
enacted by a
healthcare professional.
One of the purposes of using clinical simulation in relation to
implementation was

to investigate how the information system supported clinical
practice and to determine
whether the information system should be implemented at the
hospitals. Therefore
there was a need for high fidelity in the case study.
Figure 4. Simulation room seen from observation room.
The clinical simulation identified many uncertainties
concerning work flow,
handling of responsibility, and other organizational and
technical challenges. High
fidelity functionalities, such as integration to other information
systems, revealed
patient safety issues; e.g. notes related to a test result were not
shown in relation to the
test result in the new information system. The physician could
only find the notes in the
lab system. Apart from many negative findings, there were also
positive findings,
including improved overview of laboratory test results and no
paper test results were
left lying around, at the risk of disappearing.

We did not have any patient safety experts attending as
observers during the
simulation. Instead the simulation evaluation report was
subsequently shown to the
patient safety experts. Having patient safety experts observe the
simulation would have
improved the outcome considerably. Several organizational and
technological issues,
which were regarded as inconveniences by others, were detected
as patient safety risks
by the patient safety experts. These experts have great
experience of what can go
wrong and are able to focus on these matters during the
simulation. They observe the
interaction between the user and the interface of the technology
but just as much the
interaction with the technology in the clinical context. Inclusion
of clinical context is
one of the most powerful elements in clinical simulation. By
allowing clinicians to use
new technology in the way it is supposed to be used, patient
safety issues become
visible. Clinical simulation enables visualization of technology
in connection with

clinical context without endangering patients [22]. Therefore
the choice of observers is
very important. Each expert focuses on his or her own field. For
this reason, observers
must be chosen carefully and bearing in mind the purpose of the
simulation.
As a result of the simulation additional new requirements of the
information
system were determined, e.g. new functionality for sorting the
list of laboratory results
according to date and time of the results. It was decided to
initiate a pilot
implementation despite the fact that the information system did
not fully support the
work flows. Some of the organizational challenges were solved
and it was agreed that
the remaining challenges regarding future work practice should
be subject to scrutiny
during the pilot implementation.
The challenges not solved prior to the pilot implementation
were the transferability

of work practice between patient wards and outpatient clinics,
confidentiality of some
test results, risk of several users handling the same test result
simultaneously, missing
interaction between prescription of test and signing of test
results, no possibility of
undoing signing of test results, comments do not stand out
distinctly and integration
between information system and paper-based test results from
private laboratories. The
issues were observed and evaluated after the system was
implemented.
4. Discussion
The clinical simulation focused on formative evaluation and
primarily was used as a
learning process. Formative evaluation studies can facilitate
system adoption and
utilization [41] and aim to improve a system during its
development or implementation,
while summative evaluation focuses on evaluation of a system
that is already up and
running [42]. Formative evaluation may identify potential
problems, such as patient

safety issues, during the development phase and thus provide
opportunities to improve
a system as it develops.
In the simulation study, the results of the formative evaluation
regarding patient
safety issues and work practice for handling laboratory test
results were presented and
discussed at meetings with the various stakeholders, i.e. the
patient safety unit, the
quality unit and the implementation departments. Precautions
were taken in relation to
patient safety matters and work practice. Many of these
precautions were subsequently
implemented, regardless of the implementation of information
system.
Unintended incidents often occur in the interaction between
humans, technology
and work practice [4, 10]. Clinical simulations allow
visualization of the correlation
between human, technology and organization. More
conventional usability evaluations
tend to visualize the interaction between the user and the
technology but do not include
work practice context [20, 43]. By including all three aspects

(humans, technology and
organization), patient safety challenges were revealed as well as
organizational and
technical challenges. New work practice in itself may also lead
to unintended incidents.
This was also revealed during the clinical simulation.
To expose cognitive and socio-technical issues, all fidelity
dimensions described
by Dahl and colleagues [23] need to be high on all four
dimensions. The overall
simulation configuration affects how the realism of the
simulation experience is
perceived [38]. Cognitive aspects of work practice relate to the
clinical context and
therefore depend on the degree of environment and task realism,
whereas equipment
fidelity and functional fidelity relate to cognitive aspects of the
technical context.
Socio-technical aspects and patient safety matters lie in the
intersection between user,
organization and technology [40]. High fidelity simulations are
time-consuming [44]
though and the purpose of simulation studies and the need for
fidelity should therefore

be planned carefully.
Traditional information systems are often designed around an
idealized model of
the tasks and workflow, and failures in information systems are
often blamed on human,
social and cultural “barriers” to technology adoption [10]. The
case study revealed
differences between such an idealized model of the task that
needed to be accomplished
and the way in which clinicians were actually working. Some of
the differences were
due to local interpretations of the regional guidelines and one of
the conclusions
reached was that the regional quality unit should develop a
regional standard for
signing off test results. Another issue lay in the fact that the
information system was a
standard system which did not provide adequate opportunities to
configure the system
to match the local setting. If work practice differs from

department to department, local
configuration is a requirement. A regional standard was
introduced to resolve this issue.
Clinical simulation did not reveal all challenges related to the
information system.
The outcome of clinical simulations depends on the quality of
the scenarios and patient
cases they cover. In the case study, the scenarios during the
simulations did not include
unusual results or pre-ambulatory test results, but only became
clear during a
subsequently pilot implementation. Clinical simulation involves
an inherent risk of
giving an idealized picture compared to real life as it is very
resource demanding to
simulate the complexity of real life situations at a hospital.
These matters are important
to take into account when planning and designing the
simulation.
Another aspect is the purpose of the evaluation and the relation
between existing
and future work practice. What is to be evaluated - future or
existing work practice?
And do the end-users comprehend and approve the new work

practice? Furthermore, if
the existing work practice in a department does not follow the
existing guidelines, this
may influence the simulation of the interaction between future
work practice, end-users
and technology as well as subsequent implementation.
Several muddled work flows became clear during the simulation
and observers
focusing on work flows agreed that a further work flow analysis
was needed. This
resulted in revision of the future work practice. Many of the
issues found during the
simulation were addressed before the pilot implementation, and
those that were not
solved were observed again during the pilot implementation. As
such clinical
simulation cannot replace a pilot implementation, but should
rather be regarded as a
valuable supplement.
Patient safety issues are difficult to assess due to the fact that
many patient safety
challenges lie in the details and are triggered by adverse events
and disturbances [24].

The results of the case study showed that clinical simulation
took the clinical context
into account, while other methods, e.g. heuristic inspection,
focus on the user interface.
Low fidelity usability testing focuses on technology and
specific tasks for single users.
Some patient safety risks may therefore be difficult to pinpoint
using these methods.
Clinical simulation provides a comprehensive view on the
information system taking
into account the correlation between IT, work practice and
adverse events, and is
therefore a very suitable method for assessing patient safety
issues.
The resources invested in preparing and performing simulation
studies may be
exhaustive, depending on the required degree of fidelity. It is
essential that the
resources invested in creating a realistic setting match the
purposes of the simulation
and the simulation set-up [43, 44]. However, the resources
saved and iatrogenic effects
avoided by using clinical simulation for analysis and evaluation
purposes are difficult

to quantity as it is difficult to put a price on the value of
patients’ lives. Still, clinical
simulation is a beneficial evaluation method, as it takes place in
a controlled
environment where there is no risk of injuring real patients [20,
40].
As described clinical simulation can be used to analyse, design
and evaluate user
requirements and work practice and serve as common ground to
help to achieve a
shared understanding between various communities of practice.
The primary benefits
of using clinical simulation are 1) involvement of users and
clinical context, 2)
controlled environments for experiments and formative
evaluations of user satisfaction,
usefulness and patient safety, 3) environments for addressing
and visualizing cross-
sectorial and cross-functional topics, and 4) organizational
learning space and common
ground for gaining shared understanding.

The main concerns and challenges of using clinical simulation
are that clinical
simulation does not reflect the social-technical issues over time
and does not cover all
possible work practice situations and issues. The purpose and
choice of scenarios
determines to a great extent the outcome, and the purpose and
design of clinical
simulation must therefore be considered very carefully.
Recommended further readings
1. Kushniruk A, Nohr C, Jensen S, Borycki EM, From Usability
Testing to Clinical
Simulations: Bringing Context into the Design and Evaluation
of Usable and Safe
Health Information Technologies, Yearb Med Inform 8 (2013),
78-85.
2. Jensen S, Kushniruk A, Boundary objects in clinical
simulation and design of
eHealth, Health Informatics Journal Oct 9,
2014:1460458214551846.
3. Jensen S, Kushniruk AW, Nøhr C, Clinical simulation: A
method for development
and evaluation of clinical information systems, Journal of

Biomedical Informatics
54 (2015), 65-76.
4. Ammenwerth E, Hackl WO, Binzer K, Christoffersen TE,
Jensen S, Lawton K,
Skjoet P, Nohr C, Simulation Studies for the evaluation of
health information
technologies: experiences and results, HIM J 41(2012), 14-21.
5. Jensen S, Clinical Simulation: For what and how can it be
used in design and
evaluation of health IT, Techno-Anthropology in Health
Informatics:
Methodologies for Improving Human-Technology Relations,
Studies in Health
Technology and Informatcs 215 (2015), pp. 217-28.
Food for thought
1. What might be the pros and cons of clinical simulation seen
from an end-user
perspective and how may it differ from a management and
policy perspective?
2. Clinical simulation refers to simulation in a clinical set-up.
How may simulation fit
into other high-risk areas such as pharmacies and ambulances?
3. As healthcare technology moves into patients’ homes,
simulation could also be

used in private settings. How would a simulation design differ
when conducting
simulations in patients home?
4. How may clinical simulation be used in other clinical fields,
such as biomedical
engineering?
References
[1] E. Coiera, J. Aarts, C. Kulikowski, The dangerous decade, J
Am Med Inform Assoc 19 (2012), 2-5.
[2] D.W. Bates, M. Cohen, L.L. Leape, M.J. Overhage, M.M.
Shabot, T. Sheridan, Reducing the
frequency of errors in medicine using information technology, J
Am Med Inform Assoc 8 (2001), 299-
308.
[3] E. Ammenwerth, Evidence-Based Health Informatics: How
do we know what we know? Methods Inf
Med 54(4) (2015), 298-307.
[4] R. Koppel, T. Wetterneck, J.L. Telles, B.T. Karsh,
Workarounds to barcode medication administration
systems: their occurrences, causes, and threats to patient safety,
J Am Med Inform Assoc 15 (2008),

408-423.
[5] F. Magrabi, M. Baker, I. Sinha, M.S. Ong, S. Harrison, M.R.
Kidd, W.B. Runciman, E. Coiera,
Clinical safety of England's national programme for IT: A
retrospective analysis of all reported safety
events 2005 to 2011, International Journal of Medical
Informatics 84 (2015), 198-206.
[6] Y.Y. Han, J.A. Carcillo, S.T. Venkataraman, R.S. Clark,
R.S. Watson, T.C. Nguyen, H. Bayir, R.A.
Orr, Unexpected increased mortality after implementation of a
commercially sold computerized
physician order entry system, Pediatrics 116 (2005), 1506-1512.
[7] M.I. Harrison, R. Koppel, S. Bar-Lev, Unintended
Consequences of Information Technologies in
Health Care – An Interactive Sociotechnical Analysis, J Am
Med Inform Assoc 14 (2007), 542-549.
[8] M.C. Beuscart-Zephir, C. Nøhr, Human factors engineering
for computer-supported identification and
prevention of adverse drug events, Stud Health Technol Inform
148 (2009), 14-24.
[9] R. Koppel, D.A. Kreda, Healthcare IT usability and
suitability for clinical needs: challenges of design,
workflow, and contractual relations, Stud Health Technol
Inform 157 (2010), 7-14.
[10] E. Coeira E, Interaction design theory, International
Journal of Medical Informatics 69 (2003), 205-
222.

[11] R. Beuscart, P. McNair, S.J. Darmoni, V. Koutkias, N.
Maglaveras, M.C. Beuscart-Zephir, C. Nøhr,
Patient safety: detection and prevention of adverse drug events,
Stud Health Technol Inform 150
(2009), 968-971.
[12] YY. Han, J.A. Carcillo, S.T. Venkataraman, R.S. Clark,
R.S. Watson, T.C. Nguyen, H. Bayir, R.A.
Orr, Unexpected increased mortality after implementation of a
commercially sold computerized
physician order entry system, Pediatrics 116 (2005), 1506-1512.
[13] M. Berg, Health information management: Integrating
information and communication technology in
health care work, Routledge; 2003.
[14] P. Carayon, Human factors of complex sociotechnical
systems, Appl Ergon 37 (2006), 525-535.
[15] M. Berg, Patient care information systems and health care
work: a sociotechnical approach,
International Journal of Medical Informatics 55 (1999), 87-101.
[16] P. Carayon, E.J. Bass, T. Bellandi, A.P. Gurses, M.S.
Hallbeck, V. Mollo, Sociotechnical systems
analysis in health care: a research agenda, IIE transactions on
Healthcare Systems Engineering 1
(2011), 145-160.
[17] T.G. Kannampallil, G.F. Schauer, T. Cohen, V.L. Patel,
Considering complexity in healthcare systems,
Journal of Biomedical Informatics 44 (2011), 943-947.

[18] E.M. Borycki, M. Househ, A.W. Kushniruk, C. Kuziemsky,
Use of qualitative methods across the
software development lifecycle in health informatics, Stud
Health Technol Inform 164 (2011), 293-297.
[19] A.W. Kushniruk, V.L. Patel, Cognitive and usability
engineering methods for the evaluation of clinical
information systems, J Biomed Inform 37 (2004), 56-76.
[20] A. Kushniruk, C. Nøhr, S. Jensen, E.M. Borycki, From
Usability Testing to Clinical Simulations:
Bringing Context into the Design and Evaluation of Usable and
Safe Health Information Technologies,
Yearb Med Inform 8 (2013), 78-85.
[21] E. Borycki, A. Kushniruk, Identifying and preventing
technology-induced error using simulations:
application of usability engineering techniques, Healthc Q 8
Spec No (2005), 99-105.
[22] S. Jensen, K.M. Lyng, C. Nøhr, The role of simulation in
clinical information systems development,
Stud Health Technol Inform 180 (2012), 373-377.
[23] S. Jensen, C. Nøhr, S.L. Rasmussen, Fidelity in clinical
simulation: how low can you go? Stud Health
Technol Inform 194 (2013), 147-153.
[24] S. Jensen, S.L. Rasmussen, K.M. Lyng, Evaluation of a
Clinical Simulation-based Assessment Method
for EHR-platforms, Stud Health Technol Inform 205 (2014),
925-929.
[25] M. Berg, Health Information Systems. Integrating
information technology in health care work,

Routledge, 2006.
[26] P. Carayon, Handbook of human factors and ergonomics in
health care and patient safety, CRC Press,
2011.
[27] J.S. Ash, M. Berg, E. Coiera, Some unintended
consequences of information technology in health
care: the nature of patient care information system-related
errors, J Am Med Inform Assoc 11 (2004),
104-112.
[28] A. Kushniruk, P. Turner, A framework for user
involvement and context in the design and
development of safe e-Health systems, Stud Health Technol
Inform 180 (2012), 353-357.
[29] E.M. Borycki, L. Lemieux-Charles, L. Nagle, G.
Eysenbach, Evaluating the impact of hybrid
electronic-paper environments upon novice nurse information
seeking, Methods Inf Med 48 (2009),
137-143.
[30] C. Pearce, A. Shachak, A. Kushniruk, S. de Lusignan,
Usability: a critical dimension for assessing the
quality of clinical systems, Inform Prim Care 17 (2009), 195-
198.
[31] M. Berg, Health Information Technology in Health Care

Work, Routledge, 2004.
[32] ISO 9241-210: Ergonomics of human system interaction -
Part 210: Human-centred design for
interactive systems. International Standardization Organization
(ISO), Switzerland, 2009.
[33] M. Hertzum, Images of usability, Intl Journal of Human-
Computer Interaction 26 (2010), 567-600.
[34] M. Maguire, Methods to support human-centeres design,
Intl Journal of Human-Computer Interaction
55 (2001), 587-634.
[35] D. Robey, D. Farrow, User involvement in information
system development: A conflict model and
empirical test, Management Science 28 (1982), 73-85.
[36] L. Damodaran, User involvement in the systems design
process-a practical guide for users, Behaviour
& Information Technology 15 (1996), 363-377.
[37] S.L. Rasmussen, K.M. Lyng, S. Jensen, Achieving IT-
supported standardized nursing documentation
through participatory design, Stud Health Technol Inform 180
(2012), 1055-1059.
[38] Y. Dahl, O.A. Alsos, D. Svanæs, Fidelity Considerations
for Simulation-Based Usability Assessments
of Mobile ICT for Hospitals, International Journal of Human-
Computer Interaction 26 (2010), 445-
476.
[39] S. Jensen, S. Vingtoft, C. Nøhr, Benefits of a clinical
planning and coordination module: a simulation

study, Stud Health Technol Inform 183 (2013), 220-224.
[40] S. Jensen, Use of clinical simulation in development and
implementation of clinical information
systems, PhD thesis, University of Aalborg, 2014.
[41] H.P. Lundsgaarde, R.M. Gardner, R.L. Menlove, Using
attitudinal questionnaires to achieve benefits
optimization, in: Proceedings of the Annual Symposium of the
American Medical Informatics
Association, 1989, pp. 703-708.
[42] B. Kaplan, Addressing organizational issues into the
evaluation of medical systems, J Am Med Inform
Assoc 4 (1997), 94-101.
[43] E. Ammenwerth, S. Graber, G. Herrmann, T. Bürkle, J.
König, Evaluation of health information
systems-problems and challenges, Int J Med Inform 71 (2003),
125-135.
[44] E. Ammenwerth, W.O. Hackl, K. Binzer, T.E.
Christoffersen, S. Jensen, K. Lawton, P. Skjoet, C.
Nøhr, Simulation Studies for the evaluation of health
information technologies: experiences and
results, HIM J 41 (2012), 14-21.

Copyright of Studies in Health Technology & Informatics is the
property of IOS Press and its
content may not be copied or emailed to multiple sites or posted
to a listserv without the
copyright holder's express written permission. However, users
may print, download, or email
articles for individual use.
Title:
Assessment report for proposed researched projects
Paper type
Research Paper
Language style
English (U.S.)
Deadline
16th April 2019 @ 08:31:45 P.M. (LONDON TIME)
Paper format
APA
Course level
Undergraduate
Subject Area
Nursing
# pages
8 ( or 2200 words Minimum)
Spacing
Double Spacing
Cost
$ 40.00
# sources
4
Paper Details
A hospital marketing director at a very large hospital in an
urban setting has several research projects to undertake this
quarter. The hospital is continuing to expand its offerings in the
metropolitan area and ensure a strong relationship with the top

physicians in the community.
The 3 current projects that he is currently researching include
the following:
1) The hospital urology department wants to establish a sexual
dysfunction clinic. The department head wants to get an
estimate of the number of men ages 35-60 in the community
suffering from some form of sexual dysfunction.
2) A primary care medical group is trying to determine whether
patients are being greeted and serviced appropriately by the
billing and admitting departments.
3) A managed care organization (MCO) is trying to determine
what concerns physicians have in agreeing to become part of its
panel of physicians who will treat their enrollees.
You are assigned the following task:
Develop a report analyzing how each of these 3 projects would
be communicated to each market segment and how each will
help the hospital improve its relationship with the public.
Because of the nature of the report, it must include a critique of
current literature (scholarly references) to support a position on
each.
DO NOT PLAGIARIZE, WRITE WELL, GOOD GRAMMAR IS
A PLUS, REFERENCE WELL.
ORIGINAL PAPER
Comparison of a Self-Directed and Therapist-Assisted
Telehealth
Parent-Mediated Intervention for Children with ASD: A Pilot
RCT

Brooke Ingersoll1 • Allison L. Wainer1,2 • Natalie I. Berger1 •
Katherine E. Pickard1 •
Nicole Bonter1
Published online: 27 February 2016
� Springer Science+Business Media New York 2016
Abstract This pilot RCT compared the effect of a self-
directed and therapist-assisted telehealth-based parent-
mediated intervention for young children with ASD.
Families were randomly assigned to a self-directed or
therapist-assisted program. Parents in both groups
improved their intervention fidelity, self-efficacy, stress,
and positive perceptions of their child; however, the ther-
apist-assisted group had greater gains in parent fidelity and
positive perceptions of child. Children in both groups
improved on language measures, with a trend towards
greater gains during a parent–child interaction for the
therapist-assisted group. Only the children in the therapist-
assisted group improved in social skills. Both models show
promise for delivering parent-mediated intervention; how-

ever, therapist assistance provided an added benefit for
some outcomes. A full-scale comparative efficacy trial is
warranted.
Keywords Autism � Parent training � Telehealth
Introduction
Autism spectrum disorder (ASD) is characterized by per-
vasive deficits in social communication and the presence of
restricted and repetitive behaviors (American Psychiatric
Association 2013). Individuals with ASD often require
intensive and comprehensive intervention across the life
span (National Research Council 2001). There has been a
dramatic increase in the number of individuals with this
diagnosis over the last two decades (Fombonne 2009)
without a corresponding growth in the availability of evi-
dence-based intervention services, which has contributed to
high levels of unmet service needs for individuals with
ASD and their families (Bitterman et al. 2008; Kogan et al.
2008). Thus, systematic research examining strategies for

increasing access to evidence-based ASD services is a high
priority.
Parent-mediated intervention (PMI) programs are one
potentially cost-effective strategy to increase access to
evidence-based ASD intervention. Teaching parents to
provide intervention themselves can increase the number of
intervention hours a child receives and has been shown to
result in improvements in child social-communication
skills (e.g., Kasari et al. 2015; Wetherby et al. 2014) and
parent well-being (Keen et al. 2010). Yet, formal PMI
programs are rare in community-based settings for young
children with ASD (Thomas et al. 2007a, b). Barriers
include a shortage of trained professionals, lengthy wait-
lists, limited financial resources and transportation, lack of
child care, geographic isolation, and time limitations (Sy-
mon 2005). Thus, it is essential to consider the adaptation
of evidence-based PMI to non-traditional service delivery
models.

Telehealth, or the provision of health services and
information over the Internet and related technologies, has
the potential to replace or augment traditional service
models to increase access to evidence-based intervention
(Baggett et al. 2010; United States Department of Educa-
tion, Office of Planning, Evaluation, and Policy Develop-
ment 2010). Telehealth programs can reduce costs and
increase provider system coverage relative to traditional in-
& Brooke Ingersoll
[email protected]
1
Department of Psychology, Michigan State University, 316
Physics Rd., East Lansing, MI 48824, USA
2
Present Address: Rush University Medical Center, Chicago,
IL, USA
123
J Autism Dev Disord (2016) 46:2275–2284
DOI 10.1007/s10803-016-2755-z
http://crossmark.crossref.org/dialog/?doi=10.1007/s10803-016-

2755-z&domain=pdf
http://crossmark.crossref.org/dialog/?doi=10.1007/s10803-016-
2755-z&domain=pdf
person service delivery models (Gros et al. 2013). Such
programs have improved care for patients with chronic
diseases (Wootton 2012), and increased access to evidence-
based health promotion, psychological, and parenting
interventions (e.g., Andersson and Cuijpers 2009; Krebs
et al. 2010; Nieuwboer et al. 2013). Patients report high
levels of satisfaction with care received via telehealth
(Gustke et al. 2000) and efficacy studies have found
moderate to large effects of telehealth interventions on
participant knowledge and behavior change (Andersson
and Cuijpers; Krebs et al.; Nieuwboer et al. 2013). Further,
several meta-analyses have found that cognitive behavioral
therapy (CBT) delivered via telehealth is as effective as
traditional therapy (Andrews et al. 2010).
Telehealth interventions can be either self-directed (i.e.,
participant independently engages with the interactive

program) or therapist-assisted (i.e., participant receives
additional guidance from a professional as part of the
program). Self-directed programs have greater dissemina-
tion potential as they do not require a trained professional
and can typically be administered at a reduced cost.
However, therapist-assisted CBT telehealth programs typ-
ically lead to better patient outcomes than self-directed
programs (Andersson and Cuijpers 2009; Spek et al. 2007).
This finding may be particularly relevant for telehealth
PMIs, as research suggests that parent coaching is impor-
tant for increasing parents’ fidelity of implementation
(Kaminski et al. 2008).
Empirical evaluations of telehealth PMI programs for
children with ASD are limited. Several uncontrolled stud-
ies have demonstrated the feasibility, acceptability, and
initial effectiveness (i.e., gains in parent knowledge) of
self-directed telehealth PMI programs (Hamad et al. 2010;
Jang et al. 2012; Kobak et al. 2011). A small RCT (n = 27)

found that parents who received a DVD-based self-directed
program made greater gains in their use of pivotal response
treatment (PRT) strategies, provided more language
opportunities, and were rated as displaying greater parent
confidence during a 10-min parent–child observation in the
home than parents in a no-treatment control group (Nefdt
et al. 2009). In addition, their children demonstrated
greater gains in their rate of functional verbal utterances
during the parent–child interaction. A single-case design
study (n = 3) demonstrated that parents improved their use
of reciprocal imitation training (RIT) strategies after
completing a web-based self-directed program (Wainer and
Ingersoll 2013), and their children increased their rate of
imitative toy play during parent–child interactions.
Importantly, parents in both studies indicated that coaching
from a professional would have been beneficial.
Several single-case design studies have examined the
efficacy of therapist-assisted telehealth PMI programs for

children with ASD. Across two studies, Vismara and
colleagues examined the ability of a DVD-based (n = 8) or
web-based tutorial (n = 9) in conjunction with weekly
coaching sessions via video-conferencing to teach parents
to use the Early Start Denver Model (ESDM) (Vismara
et al. 2012, 2013). Parents in both studies increased their
use of the ESDM intervention strategies and positively
altered their engagement styles during parent–child inter-
actions in response to the program. Furthermore, their
children demonstrated gains in functional verbal utterances
and imitative play actions. To evaluate the relative con-
tributions of self-directed instruction and therapist assis-
tance on parent learning, Wainer and Ingersoll (2015)
conducted a second study (n = 5) that measured parents’
use of RIT at baseline, after a self-directed web-based
tutorial, and then again after receiving coaching via video-
conferencing. Similar to their previous study (Wainer and
Ingersoll 2013), all parents improved their use of RIT

strategies during parent–child interactions in response to
the self-directed program, but roughly a third showed
additional benefit from remote coaching. In addition, most
children demonstrated gains in imitative play with the
onset of treatment; however, child gains were most robust
when parents received coaching.
These preliminary findings suggest telehealth PMI pro-
grams are acceptable to parents of children with ASD, and
can improve parent knowledge and intervention use and
child social communication skills. To date, no studies have
conducted a head-to-head comparison of self-directed and
therapist-assisted telehealth PMI. Data thus far suggest
therapist assistance may be necessary for some, but not all
parents to implement interventions with fidelity. A better
understanding of the contributions of self-directed
instruction and therapist assistance will make it possible to
develop more cost-effective delivery models in which
services are offered at different levels of intensity,

depending on family needs (i.e., stepped-care) (Phaneuf
and McIntyre 2011; Steever 2011). Pilot studies enhance
the likelihood of success of randomized controlled trials
(RCT) (e.g., Campbell et al. 2007). Thus, the goal of this
pilot study was to compare the effect of self-directed and
therapist-assisted delivery models of ImPACT Online, a
telehealth PMI program that targets social communication
using a naturalistic, developmental-behavioral intervention
(NDBI) (Schreibman et al. 2015), on key parent and child
outcomes in preparation for a fully powered RCT.
Method
Participants
Twenty-eight families of a child with ASD between the
ages of 19 and 73 months participated. Participants were
2276 J Autism Dev Disord (2016) 46:2275–2284
123
recruited from agencies serving children with ASD. All

children met criteria for Autistic Disorder or PDD-NOS
based on DSM-IV criteria (American Psychiatric Associ-
ation 2000) and the ADOS-G or ADOS-2 (Lord et al.
2000). Parents had to be proficient in English, although
other languages could be spoken in the home. One family
who was assigned to the therapist-assisted group had to
suspend treatment after 2 months due to significant health
problems. After a 7-month break, the family finished the
program; however, the time between their pre- and post-
treatment assessments (14 months) was not comparable to
the other participants’. Thus, their data were not included
in the analyses. All parents gave informed consent under
the oversight of Michigan State University’s IRB.
Design and Procedure
Families were administered standardized assessments at
pre-treatment in the lab and family home. Children were
matched within 3 months on expressive language age on
the Mullen Scales of Early Learning (MSEL; Mullen 1995)

and randomly assigned to the self-directed (n = 13) or
therapist-assisted group (n = 14) using a coin flip.
Assessments were re-administered at post-treatment and
3-month follow-up in the family home. See Table 1 for
participant flow through the study.
Eligibility and Sample Characteristics Measures
Parents provided information on family and child demo-
graphics at pre-treatment. They also provided information
on hours per week of all non-study treatments at pre- and
post-treatment. Children were administered the ADOS-G
or ADOS-2 to determine study eligibility and the MSEL as
a measure of developmental functioning. See Table 2 for
participant demographic information.
Parent Outcomes Measures
Parent Intervention Fidelity
Parents were videotaped during a parent–child interaction
(PCI) in the family home at pre-treatment, post-treatment,
and 3-month follow-up. They were asked to: (1) play with

Table 1 Participant flow
through the study Assessed for eligibility (n=29)
Excluded (n=1)
Not meeting inclusion criteria (n=1)
Declined to participate (n=0)
Other reasons (n=0)
Analysed (n=13)
Excluded from analysis (n=0)
Lost to follow-up (n=0)
Discontinued intervention (did not have time;
family crisis; began intensive program) (n=4)
Allocated to self-directed group (n=13)
Received allocated intervention (n=13)
Did not receive allocated intervention (n=0)
Lost to follow-up (n=0)
Discontinued intervention (n=0)
Allocated to therapist-assisted group (n=15)
Received allocated intervention (n=15)
Did not receive allocated intervention (n=0)
Analysed (n=14)
Excluded from analysis (family took 14
months to complete due to crisis) (n=1)
Allocation
Analysis

Follow-Up
Randomized (n=28)
Enrollment
J Autism Dev Disord (2016) 46:2275–2284 2277
123
their child for 10 min; and (2) have a snack with their
child. Parent behavior was scored for correct use of the
intervention using the Project ImPACT fidelity checklist
(Ingersoll and Dvortcsak 2010). Fidelity ratings for the
play and snack routines were averaged to form an overall
fidelity rating for each time point. Reliability was calcu-
lated for 20 % of the observations using intra-class corre-
lation (ICC = .96, p .001).
Parent Sense of Competence Scale (PSOC)
Parents completed the PSOC (Gibaud-Wallston and Wan-
dersmann 1978) at pre- and post-treatment as a measure of
parent self-efficacy. Parents rated items from 1 (‘‘Strongly

agree’’) to 6 (‘‘Strongly disagree’’) with higher scores
indicative of higher parenting self-efficacy. Missing item
level data (5 %) were imputed by using the participant’s
average for the scale. Cronbach’s alpha was .85.
Family Impact Questionnaire (FIQ)
Parents completed the FIQ (Donenberg and Baker 1993) at
pre- and post-treatment as a measure of the impact of their
child on their family life. Parents endorse items on a
4-point scale, with higher scores indicating greater impact.
The average of the negative and social impact subscales
was used as a measure of parenting stress, and the positive
impact subscale was used as the measure of positive per-
ceptions of the child. Missing item level data (5 %) were
imputed by using the participant’s average for the scale.
Cronbach’s alphas ranged from .86 to .92 for the parenting
stress items and from .81 to .86 for the positive impact
subscale.
Child Outcome Measures
Language Targets

The children’s use of language targets was scored during
the PCI at pre, post, and follow-up. Language that was at or
above the child’s targeted language level and used appro-
priately was scored using frequency counts and converted
to rate per minute. Language targets were determined at
intake based on the child’s spontaneous language during
standardized and observational assessments using Tager-
Flusberg et al. (2009) framework for defining spoken lan-
guage benchmarks for children with ASD. Language acts
were scored if they were at or above the child’s current
expressive language phase, which included word approxi-
mations for children at the preverbal communication stage,
single words for children at the first words stage, phrase
speech for children at the word combinations stage, and
grammatically correct sentences for children at the sen-
tences stage. Both prompted and spontaneous use of lan-
guage targets was scored. Data from the play and snack
routines were averaged to form an overall rate of language

targets for each time point. Reliability was calculated for
25 % of the observations (ICC = .98, p .001).
Table 2 Participant demographic information
Group Test statistic p value
Self-directed (n = 13) Therapist-assisted (n = 14)
Parent characteristics
Gender (% female) 92 % 100 % 1.12
a
.29
Education (% less than college degree) 54 % 36 % .90
a
.34
Marital status (% not married) 8 % 29 % 1.95
a
.16
Employment status (% not employed) 46 % 29 % .65
a
.42
Residence in underserved area 77 % 64 % .52
a

.47
Child characteristics
Gender (% female) 39 % 21 % .94
a
.33
Race/ethnicity (% minority) 8 % 36 % 3.06
a
.08
Chronological age (Mos.) 46.08 (13.18) 41.57 (12.24) -.71
b
.48
Nonverbal mental age (Mos.) 25.42 (13.92) 24.29 (9.38) -.25
b
.80
Verbal mental age (Mos.) 19.15 (9.63) 21.64 (10.74) .63
b
.53
Outside intervention (Hrs/wk)
c
13.62 (10.96) 12.38 (9.70) -.31
b
.76

a
Chi-square
b
t test
c
Average of hours received at pre- and post-treatment
123
MacArthur-Bates Communicative Development Inventory
(MCDI)
Parents completed the MCDI at pre and post as a measure
of their child’s expressive vocabulary (Fenson et al. 2006).
The total number of words reported as ‘‘understands and
says’’ (Words and Gestures Form) or ‘‘words produced’’
(Word and Sentences Form) was used. Test–retest relia-
bility of the MCDI is .95 (Fenson et al. 1994). The MCDI
was missing at pre-treatment for one participant.
Vineland Adaptive Behavior Scales, Second Edition

(VABS-II)
Parents were interviewed on the VABS-II at pre and post
(Sparrow et al. 2005). The VABS-II is a standardized
parent interview that assesses child adaptive functioning in
four domains: Communication, Socialization, Daily Living
Skills, and Motor Skills. Each domain yields a standard
score with a mean of 100 and a standard deviation of 15.
Test–retest reliability estimates range from .88 to .92 for
the domain scores. Given the focus of ImPACT Online
(social communication development), we expected
improvement in standard scores on the Social and Com-
munication domains indicating accelerated development.
As a comparison, we examined changes on the Daily
Living and Motor Skills domains, with no expectation for
improvement. The VABS was missing at post-treatment for
one participant.
Group Assignment
Self-Directed Group

The self-directed group received access to the secure,
password-protected, ImPACT Online website for
6 months. The content was adapted from Project ImPACT
(Ingersoll and Dvortcsak 2010), a NDBI-based PMI for
young children with ASD targeting social communication
development. The website contained 12, self-directed les-
sons; each took approximately 75 min to complete. Parents
were encouraged to complete one lesson per week and to
practice the intervention with their child between each
lesson. Parents were able to contact project staff for
assistance with technology-related problems, but received
no staff support in learning the intervention. See Table 3
for description of the ImPACT Online program
components.
Therapist-Assisted Group
The therapist-assisted group was given access to the
ImPACT Online website for 6 months and was encouraged
to work through the program at the same pace as the self-

directed group. These parents also received 2, 30 min
coaching sessions per week (24 total) from a trained ther-
apist via video-conferencing. The first coaching session of
each week was used to clarify the lesson content and help
parents apply the information to their child. The second
Table 3 ImPACT online program components
Primary lesson components
Slideshow Users watch narrated slideshow with embedded video
examples of techniques
Manual Users read the manual which provides a written
description of lesson that corresponds to slideshow
Self-check Users answer comprehension check questions based
on content of slideshow. The program provides automated
positive
and corrective feedback
Exercises Users observe brief video clips and must indicate
whether technique is implemented correctly or incorrectly. The
program provides automated positive and corrective feedback
Homework Users complete a homework plan that outlines
techniques and activities in which to practice. These responses
are
available to the coach for the therapist-assisted group

Reflection Users complete reflection questions based on their
practice. These responses are available to the coach for the
therapist-
assisted group
Supplemental components
Video library Users can observe longer videos of adults using
the intervention techniques together with children at different
language
levels
Forum Users can share information with other participants and
post content-related questions and receive feedback from
project
staff
Resources Users can access paper copies of all forms, additional
information on the evidence-base for this intervention, and links
to
relevant websites
Tip of the week
Emails
Users receive weekly ‘‘Tip of the Week’’ emails that provide
tips for implementing the intervention techniques along
with a link to the program

123
session of the week was used to provide parents with live
feedback on their intervention use with their child.
Coaching was provided by masters’ level therapists trained
to fidelity. Coaches used a fidelity checklist to rate their
fidelity to the coaching procedure at the end of each ses-
sion. Average self-assessed fidelity across sessions was
99.6 %. A random sample of 10 % of coaching sessions
were scored by independent raters for reliability using
exact agreement. Reliability for coaching fidelity was
97.8 %.
Results
Analytic Strategy
We used an Intention-to-Treat model (ITT) which requires
that all randomly assigned participants be compared on
outcomes regardless of adherence to treatment, reasons for

withdrawal, or missing responses (Moher et al. 2001).
Accordingly, we included all participants in the data
analysis, followed up participants who discontinued treat-
ment, and imputed missing data. Four parents, all in the
self-directed group, did not complete the program, which
represented a significant group difference in rate of pro-
gram completion, v2 (n = 27) = 5.06, p = .03. We
obtained post-treatment data for these families on all
measures, with the exception of the VABS-II and PCI for
one family. The MCDI was missing for one child in the
therapist-assisted group at pre-treatment. We used partici-
pants’ data from the opposite point of treatment for the
missing data point. We collected follow-up 3-month PCI
data from the 23 parents who completed the program. For
these data, we conducted a completer analysis only.
Transformations were applied to non-normal data as
appropriate. VABS-II data remained non-normal after
multiple transformation attempts, potentially due to out-
liers. We report analyses for the VABS-II with non-normal

data as ANOVA is robust to this violation.
Independent t tests and Chi-square tests revealed no
statistically significant group differences on any pre-treat-
ment variables, although a marginally significant group
difference (p = .08) was observed in the percent of chil-
dren who were minorities. To determine the effect of the
intervention on parent and child outcomes, we conducted a
series of mixed-model repeated measures ANOVAs with
time (pre, post) as the within group variable and group
(self-directed, therapist-assisted) as the between group
variable. We conducted a second set of mixed-model
repeated measures ANOVAs on the follow-up PCI data
with time (pre, follow-up) and group as the within and
between group variables. Post hoc comparisons of simple
effects of significant interactions were conducted using
relevant t tests. See Table 4 for ANOVA results for out-
come variables.
Parent Outcomes

There was a significant main effect of time for parent
fidelity, such that parents were rated higher on their use of
the intervention at post-treatment. There was also a sig-
nificant time X group interaction, indicating the therapist-
assisted group made greater gains in fidelity than the self-
directed group. Post-hoc tests of simple effects suggested
Table 4 ANOVA results for outcome variables
Outcome measure Self-directed group Therapist-assisted group
Effects
Pre Post Pre Post Time Time X group
Mean (SD) Mean (SD) Mean (SD) Mean (SD) F (gp
2
) F (gp
2
)
Parent fidelity (PCI) 1.77 (.67) 2.52 (.78) 1.62 (.37) 3.39 (.76)
65.78** (.72) 10.76** (.30)
Parent self-efficacy (PSOC) 53.23 (13.14) 58.62 (12.12) 54.60
(14.81) 61.43 (13.27) 10.98** (.31) .15 (.006)
Parenting stress (FIQ) 1.24 (.84) 1.04 (.61) 1.02 (.46) .69 (.30)
6.53* (.21) 2.18 (.08)

Parent positive perceptions (FIQ) 1.52 (.58) 1.60 (.50) 1.40
(.72) 2.06 (.56) 13.31** (.35) 8.27** (.25)
Child language targets (PCI) 1.36 (1.53) 1.95 (1.08) .65 (.52)
1.80 (1.00) 31.33** (.56) 2.84
?
(.10)
Child vocabulary (MCDI) 144.69 (146.21) 210.38 (187.46)
185.114 (202.17) 243.64 (237.94) 18.53** (.43) 1.17 (.05)
Child communication (VABS-II) 71.50 (15.57) 75.33 (12.40)
70.29 (11.28) 77.36 (13.79) 6.10* (.20) .68 (.03)
Child social skills (VABS-II) 71.00 (8.05) 70 .00 (7.56) 70.00
(6.95) 75.71 (9.07) 2.58 (.09) 4.94* (.17)
Child daily living skills (VABS-
II)
74.77 (13.79) 74.23 (10.42) 75.07 (7.77) 77.00 (11.14) .26 (.01)
.83 (.03)
Child motor skills (VABS-II) 81.85 (11.47) 82.85 (9.74) 83.36
(8.87) 83.14 (11.27) .08 (.003) .18 (.007)
** .01; * .05; ? .10
123
both groups had higher fidelity ratings at post- than pre-

treatment (ps .01), and at post-treatment, the therapist-
assisted group was significantly higher than the self-di-
rected group (p .01). At follow-up, there was a signifi-
cant main effect of time, F(1, 21) = 44.26, p .001,
gp
2
= .68, suggesting that the benefits of the program on
parent fidelity maintained. The time X group interaction
was not significant, F(1, 21) = 1.17, p = .29, gp
2
= .05,
suggesting that the therapist-assisted group (M = 3.00,
SD = .66) no longer showed an advantage over the self-
directed group (M = 2.57, SD = 1.21) at follow-up,
although their mean fidelity ratings remained higher.
There was a significant main effect of time on parent
self-efficacy, such that parents in both groups rated them-
selves as more efficacious at post-treatment. There was no
significant effect of group or time X group interaction.
There was a main effect of time on parenting stress, such
that parents in both groups rated themselves as less stressed

at post-treatment, but no significant main effect of group or
time X group interaction. Finally, there was a significant
main effect of time and time X group interaction for pos-
itive perceptions of the child. Post-hoc tests suggested that
the therapist-assisted group had significant (p = .001) and
the self-directed group had marginally significant (p = .09)
increases in their positive perceptions of their child, and the
therapist-assisted group had significantly more positive
perceptions than the self-directed group at post-treatment
(p = .03).
Child Outcomes
There was a significant main effect of time on language
targets during the PCI, indicating improvement in child
language use over time. There was also a marginally sig-
nificant time X group interaction (p = .10), suggesting that
the children in the therapist-assisted group made margin-
ally more gains in their use of language targets than the
self-directed group. Post-hoc tests indicated children in

both groups exhibited significant gains in their language
targets from pre- to post-treatment (ps .05), but there
were no differences between groups at post-treatment. At
follow-up, there was a significant main effect of time on
child language targets, F(1, 21) = 8.59, p = .008,
gp
2
= .29, and a marginally significant time X group
interaction, F(1, 21) = 3.38, p = .08, gp
2
= .14. Post-hoc
tests indicated language targets were higher at follow-up
(M = 1.71, SD = 1.28) than pre-treatment for the thera-
pist-assisted group (p = .003). The difference in language
targets from pre-treatment (M = 1.50, SD = 1.81) to fol-
low-up (M = 1.64, SD = 1.21) was not significant for the
self-directed group, although the trend was in the expected
direction. There were no group differences in language
targets at follow-up.
On the MCDI and VABS-II Communication domain,
there was a main effect of time, but no effect of group or

time X group interaction, suggesting that children in both
groups improved their language skills on these measures
from pre- to post-treatment. On the VABS-II Social
domain, there were no main effects of time or group;
however, there was a significant time X group interaction.
Post-hoc tests suggested that the children in the therapist-
assisted group exhibited significant increases in their
standard scores (p = .04), while the children in the self-
directed group did not. At post-treatment, the therapist-
assisted group had marginally higher standard scores than
the self-directed group (p = .08).
No significant main effects of time or group and no
significant time X group interactions were observed for the
VABS-II Daily Living or Motor Skills domains.
Discussion
This pilot RCT compared the effect of a self-directed and a
therapist-assisted telehealth PMI for children with ASD on
parent and child outcomes. Parents in both groups

improved their fidelity. This finding suggests the self-di-
rected website was sufficient for increasing parent inter-
vention use and provides additional support for the benefits
of self-directed telehealth programs for increasing parents’
skills (Nefdt et al. 2009; Wainer and Ingersoll 2013).
However, parents in the therapist-assisted group made
greater gains in their use of the intervention. Similar
findings were observed regarding parents’ perceptions of
their child; parents in the therapist-assisted group had
greater increases in their positive perceptions of their child
than the self-directed group.
These findings are consistent with research on the ben-
efits of coaching in traditional PMI (Kaminski et al. 2008),
and suggest that coaching may be important for maximiz-
ing parent gains from telehealth PMI as well. Coaching
provides parents with feedback on their intervention use
and highlights its impact on their child’s behavior. Parents
who received coaching likely thus developed a greater

understanding of their child’s skills and a better apprecia-
tion for the impact of their own behavior on their child’s,
potentially improving their positive perceptions of their
child. Higher rates of program completion among the
therapist-assisted group may also have contributed to
observed group differences. Indeed, these parents were
significantly more likely to complete the program (100 %)
than parents in the self-directed group (69 %). Thus,
therapist assistance may be important not only for pro-
viding parents with feedback on their use of the interven-
tion and highlighting child skills, but also for encouraging
higher levels of engagement with the program in general,
123
both of which may influence parent fidelity and/or positive
child perceptions. Given that program completion was
related to increases in parent fidelity (Ingersoll and Berger

2015), this is a distinct possibility. Either way, there
appears to be a benefit of therapist assistance on these two
parent outcomes. Additional research that can examine
which aspects of therapist assistance are most important
would be beneficial.
Improvements were also observed for parent self-effi-
cacy and parenting stress. This finding is important as
parents of children with ASD often experience lower self-
efficacy and higher stress than other parents (Hayes and
Watson 2013), and both are related to parent well-being
(Carter 2009; Karst and Hecke 2012). Interestingly, there
were no group differences in these parent outcomes. These
findings suggest that the self-directed program may be
sufficient for increasing parent well-being. Perhaps parents
who receive instruction in strategies to help their child
experience a greater sense of empowerment or agency,
regardless of instructional format. Indeed, parent empow-
erment and agency are associated with greater self-efficacy

and lower parenting stress in families of children with ASD
(Kuhn and Carter 2006). Future research that can identify
those components most likely to improve parent self-effi-
cacy and parenting stress would be informative. We also
found evidence for improvement in children’s social
communication skills. Although the groups did not differ in
the degree of improvement on the two parent-reported
language measures, the children in the therapist-assisted
group made marginally greater gains in their language
targets during the PCI than the self-directed group. These
findings suggest that both the self-directed and therapist-
assisted formats may be effective for increasing child
communication skills, although there may be a small
benefit of therapist assistance, as observed within the PCI.
In contrast, only the therapist-assisted group made
improvements in their social skills on the VABS-II. Studies
examining parent-mediated social communication inter-
ventions for young children with ASD have more often

reported improvements in child language than social skills
(McConachie and Diggle 2007). Social skills may be less
amenable to treatment than language, and as such, therapist
assistance may be necessary to help parents improve their
child’s social development. For this study, we had only one
broad measure of social skills. Thus, it is possible that we
would have observed improvements in the self-directed
group as well, had we used a more sensitive measure or
examined specific skills. Future research that can determine
which social skills are most responsive to the intervention,
as well as the potential benefit of the self-directed program
on specific social skills is necessary.
A major limitation to this study is the small sample size
inherent in a pilot study. Our small sample and resultant
limited power may have contributed to our failure to
identify group differences in treatment response for several
of our outcome measures. For example, despite not
reaching significance, the effect sizes for the time X group

interactions for parenting stress (gp
2
= .08) and child lan-
guage targets (gp
2
= .10) were in the moderate range,
suggesting that with a larger sample size, these interactions
would likely have become significant. A power analysis
suggested that we would have needed a sample size of 60
to detect significant differences for effect sizes in this
range.
In addition, while we were able to establish an effect of
therapist assistance for those outcomes for which the
therapist-assisted group demonstrated greater improvement
than the self-directed group, without a no-treatment control
group, we cannot conclude that the improvements observed
in the self-directed group were a result of treatment. Sim-
ilarly, we cannot rule out maturation or placebo effects for
those outcomes for which both groups improved to the
same degree. We were, however, able to compare child

changes across VABS-II domains. As predicted, we found
improvements for only those domains targeted by the
program, suggesting that maturation or a placebo effect
(i.e., parent reporting bias) were unlikely solely responsible
for gains in child skills. However, additional research that
includes a web-based control is necessary to establish the
effect of both formats on parent and child functioning.
One additional limitation is that our treatment groups
differed in number of children from minority backgrounds,
with a larger number of minority children in the therapist-
assisted group (36 %) than the self-directed group (8 %). It
is not clear to what extent this might have impacted the
findings. Child minority status was not associated with rate
of parent program engagement (Ingersoll and Berger 2015)
for either group. However, it is possible that it might
impact other aspects of the treatment process. Future
research should examine the degree to which racial/ethnic
background and other demographic variables might affect

treatment response.
These data contribute to the growing empirical support
for the benefits of parent-mediated NDBIs on parent and
child outcomes (Schreibman et al. 2015), and provide
further support for the ability of telehealth to teach parents
to deliver these interventions (Nefdt et al. 2009; Vismara
et al. 2013; Wainer and Ingersoll 2015). These pilot data
suggest a potential role for both self-directed and therapist-
assisted programs for increasing parent access to evidence-
based interventions, and suggest the potential benefit of
supplementing telehealth interventions with therapist sup-
port. In addition, these data strongly support conducting a
fully powered comparative efficacy trial to examine
mediators and moderators of treatment. Such a study
should identify which outcomes are best targeted by each
123

format, as well as which parents and children benefit from
the different levels of care. The ultimate goal is to develop
a stepped-care model that can increase access to evidence-
based interventions, and reduce the high level of unmet
needs experienced by families of children with ASD.
Acknowledgments This project was supported by CDMRP
Grant:
#W81XWH-10-1-0586. The first author receives royalties from
the
sale of the manual that was adapted for use in the online
tutorial.
Royalities are donated to the research. We thank the families
who
participated in this study.
Author Contributions BI conceived of the study, participated in
its
design and coordination and drafted the manuscript; ALW, NIB,
and
KEP performed the measurement and participated in its design
and
coordination and helped to draft the manuscript. NB performed
the
measurement and participated in the study’s coordination. All

authors
read and approved the final manuscript.
References
American Psychiatric Association. (2000). DSM-IV. Diagnostic
and
statistical manual of mental disorders 4th edition TR. Arlington,
VA: American Psychiatric Publishing.
American Psychiatric Association. (2013). Diagnostic and
statistical
manual of mental disorders (5th ed.). Arlington, VA: American
Psychiatric Publishing.
Andersson, G., & Cuijpers, P. (2009). Internet-based and other
computerized psychological treatments for adult depression: A
meta-analysis. Cognitive Behaviour Therapy, 38(4), 196–205.
Andrews, G., Cuijpers, P., Craske, M. G., McEvoy, P., & Titov,
N.
(2010). Computer therapy for the anxiety and depressive
disorders is effective, acceptable and practical health care: A
meta-analysis. PLoS one, 5(10), e13196.
Baggett, K. M., Davis, B., Feil, E. G., Sheeber, L. B., Landry,

S. H.,
Carta, J. J., & Leve, C. (2010). Technologies for expanding the
reach of evidence-based interventions: Preliminary results for
promoting social-emotional development in early childhood.
Topics in Early Childhood Special Education, 29(4), 226–238.
Bitterman, A., Daley, T. C., Misra, S., Carlson, E., &
Markowitz, J.
(2008). A national sample of preschoolers with autism spectrum
disorders:Specialeducationservicesandparentsatisfaction.
Journal
of Autism and Developmental Disorders, 38(8), 1509–1517.
Campbell, N. C., Murray, E., Darbyshire, J., Emery, J., Farmer,
A.,
Griffiths, F., et al. (2007). Designing and evaluating complex
interventions to improve health care. British Medical Journal,
334, 455–459.
Carter, A. (2009). Stability and individual change in depressive
symptoms among mothers raising young children with ASD:
Maternal and child correlates. Journal of Clinical Psychology,
65(12), 1270–1280.

Donenberg, G., & Baker, B. L. (1993). The impact of young
children
with externalizing behaviors on their families. Journal of
Abnormal Child Psychology, 21(2), 179–198.
Fenson, L., Dale, P. S., Reznick, J. S., Bates, E., Thal, D., &
Pethick,
S. (1994). Variability in early communicative development.
Monographs of the Society for Research in Child Development,
59, 1–173.
Fenson, L., Marchman, V. A., Thal, D., Dale, P. S., Reznick, J.
S., &
Bates, E. (2006). MacArthur-Bates communicative development
inventories: User’s guide and technical manual 2. Baltimore,
MD: Brookes Publishing.
Fombonne, E. (2009). Epidemiology of pervasive developmental
disorders. Pediatric Research, 65(6), 591–598.
Gibaud-Wallston, J., & Wandersmann, L. P. (1978).
Development and
utility of the parenting sense of competence scale. Washington:
John F. Kennedy Center for Research on Education and Human

Development.
Gros, D. F., Morland, L. A., Greene, C. J., Acierno, R.,
Strachan, M.,
Egede, L. E., & Frueh, B. C. (2013). Delivery of evidence-based
psychotherapy via video telehealth. Journal of Psychopathology
and Behavioral Assessment, 35(4), 506–521.
Gustke, S. S., Balch, D. C., West, V. L., & Rogers, L. O.
(2000).
Patient satisfaction with telemedicine. Telemedicine Journal,
6(1), 5–13.
Hamad, C. D., Serna, R. W., Morrison, L., & Fleming, R.
(2010).
Extending the reach of early intervention training for practici-
tioners: A preliminary investigation of an online vurriculum for
teaching behavioral intervention knowledge in autism to
families
and service providers. Infants and Young Children, 23(3),
195–208.
Hayes, S., & Watson, S. (2013). The impact of parenting stress:
A meta-

analysis of studies comparing the experience of parenting stress
in
parents of children with and without autism spectrum. Journal
of
Autism and Developmental Disorders, 43(3), 629–642.
Ingersoll, B., & Berger, N. I. (2015). Parent engagement with a
telehealth-based parent-mediated intervention program for chil-
dren with ASD: Predictors of program use and parent outcomes.
Journal of Medical Internet Research, 17, e227.
Ingersoll, B., & Dvortcsak, A. (2010). Teaching social
communica-
tion: A practitioner’s guide to parent training for children with
autism. New York: Guilford Press.
Jang, J., Dixon, D. R., Tarbox, J., Granpeesheh, D., Kornack, J.,
& de
Nocker, Y. (2012). Randomized trial of an eLearning program
for training family members of children with autism in the
principles and procedures of applied behavior analysis.
Research
in Autism Spectrum Disorders, 6(2), 852–856.
Kaminski, J. W., Valle, L. A., Filene, J. H., & Boyle, C. L.

(2008). A
meta-analytic review of components associated with parent
training program effectiveness. Journal of Abnormal Child
Psychology, 36(4), 567–589.
Karst, J., & Hecke, A. Van. (2012). Parent and family impact of
autism spectrum disorders: A review and proposed model for
intervention evaluation. Clinical Child and Family Psychology
Review, 5(3), 247–277.
Kasari, C., Gulsrud, A., Paparella, T., Hellemann, G., & Berry,
K.
(2015). Randomized comparative efficacy study of parent-
mediated interventions for toddlers with autism. Journal of
Consulting and Clinical Psychology, 83(3), 554–563.
Keen, D., Couzens, D., Muspratt, S., & Rodger, S. (2010). The
effects of
a parent-focused intervention for children with a recent
diagnosis
of autism spectrum disorder on parenting stress and
competence.
Research in Autism Spectrum Disorders, 4(2), 229–241.

Kobak, K. A., Stone, W. L., Wallace, E., Warren, Z., Swanson,
A., &
Robson, K. (2011). A web-based tutorial for parents of young
children with autism: Results from a pilot study. Telemedicine
Journal and E-Health: The Official Journal of the American
Telemedicine Association, 17(10), 804–808.
Kogan, M. D., Strickland, B. B., Blumberg, S. J., Singh, G. K.,
Perrin,
J. M., & Van Dyck, P. C. (2008). A national profile of the
health
care experiences and family impact of autism spectrum disorder
among children in the United States, 2005–2006. Pediatrics,
122(6), e1149–e1158.
Krebs, P., Prochaska, J. O., & Rossi, J. S. (2010). Defining
what
works in tailoring: A meta-analysis of computer tailored
interventions for health behavior change. Preventive Medicine,
51(3–4), 214–221.
123

Kuhn, J., & Carter, A. (2006). Maternal self-efficacy and
associated
parenting cognitions among mothers of children with autism.
American Journal of Orthopsychiatry, 76(4), 564–575.
Lord, C., Risi, S., Lambrecht, L., Cook, E. H, Jr, Leventhal, B.
L.,
DiLavore, P. C., & Rutter, M. (2000). Autism diagnostic
observation schedule—generic: A standard measure of social
and communication deficits associated with the spectrum of
autism. Journal of Autism and Developmental Disorders, 30(3),
205–223.
McConachie, H., & Diggle, T. (2007). Parent implemented early
intervention for young children with autism spectrum disorder:
A
systematic review. Journal of Evaluation in Clinical Practice,
13, 120–129.
Moher, D., Schulz, K. F., Altman, D. G., & Consort Group.
(2001).
The CONSORT statement: revised recommendations for

improving the quality of reports of parallel-group randomised
trials. The Lancet, 357(9263), 1191–1194.
Mullen, E. M. (1995). Mullen scales of early learning. Circle
Pines,
MN: AGS.
National Research Council. (2001). Educating children with
autism.
Washington, DC: National Academy Press.
Nefdt, N., Koegel, R., Singer, G., & Gerber, M. (2009). The use
of a
self-directed learning program to provide introductory training
in
pivotal response treatment to parents of children with autism.
Journal of Positive Behavior Interventions, 12(1), 23–32.
Nieuwboer, C. C., Fukkink, R. G., & Hermanns, J. M. A.
(2013).
Online programs as tools to improve parenting: A meta-analytic
review. Children and Youth Services Review, 35(11),
1823–1829.
Phaneuf, L., & McIntyre, L. L. (2011). The application of a
three-tier

model of intervention to parent training. Journal of Positive
Behavior Interventions, 13(4), 198–207.
Schreibman, L., Dawson, G., Stahmer, A. C., Landa, R., Rogers,
S. J.,
McGee, G. G., & Halladay, A. (2015). Naturalistic develop-
mental behavioral interventions: Empirically validated treat-
ments for autism spectrum disorder. Journal of Autism and
Developmental Disorders, 45, 2411–2428.
Sparrow, S. S., Balla, D. A., & Cicchetti, D. V. (2005).
Vineland II:
Vineland adaptive behavior scales. Circle Pines, MN: American
Guidance Service.
Spek, V., Cuijpers, P., Nyklı́cek, I., Riper, H., Keyzer, J., &
Pop, V.
(2007). Internet-based cognitive behaviour therapy for
symptoms
of depression and anxiety: A meta-analysis. Psychological
Medicine, 37(3), 319–328.
Steever, M. (2011). Stepped care and e-health: Practical
applications
to behavioral disorders. In W. T. O’Donohue & C. Drapper

(Eds.), Autism. New York, NY: Springer.
Symon, J. B. (2005). Expanding interventions for children with
autism: Parents as trainers. Journal of Positive Behavior
Interventions, 7(3), 159–173.
Tager-Flusberg, H., Rogers, S., Cooper, J., Landa, R., Lord, C.,
Paul,
R., & Yoder, P. (2009). Defining spoken language benchmarks
and selecting measures of expressive language development for
young children with autism spectrum disorders. Journal of
Speech, Language, and Hearing Research, 52(3), 643–652.
Thomas, K. C., Ellis, A. R., McLaurin, C., Daniels, J., &
Morrissey, J.
P. (2007a). Access to care for autism-related services. Journal
of
Thomas, K. C., Morrissey, J. P., & McLaurin, C. (2007b). Use
of
autism-related services by families and children. Journal of
United States Department of Education, Office of Planning,

Evalu-
ation, and Policy Development. (2010). Evaluation of evidence-
based practices in online learning: A meta-analysis and review
of online learning studies. Washington, DC: Author.
Vismara, L. A., McCormick, C., Young, G. S., Nadhan, A., &
Monlux, K. (2013). Preliminary findings of a telehealth
approach
to parent training in autism. Journal of Autism and Develop-
mental Disorders, 43(12), 2953–2969.
Vismara, L. A., Young, G. S., & Rogers, S. J. (2012).
Telehealth for
expanding the reach of early autism training to parents. Autism
Research and Treatment, 2012, 121878.
Wainer, A. L., & Ingersoll, B. R. (2013). Disseminating ASD
interventions: A pilot study of a distance learning program for
parents and professionals. Journal of Autism and Developmental
Disorders, 43(1), 11–24.
Wainer, A. L., & Ingersoll, B. (2015). Increasing access to an
ASD
imitation intervention via a telehealth parent training program.

Journal of Autism and Developmental Disorders, 45,
3877–3890.
Wetherby, A. M., Guthrie, W., Woods, J., Schatschneider, C.,
Holland, R. D., Morgan, L., & Lord, C. (2014). Parent-
implemented social intervention for toddlers with autism: An
RCT. Pediatrics, 134(6), 1084–1093.
Wootton, R. (2012). Twenty years of telemedicine in chronic
disease
management—an evidence synthesis. Journal of Telemedicine
and Telecare, 18(4), 211–220.
123
Journal of Autism & Developmental Disorders is a copyright of
Springer, 2016. All Rights
Reserved.
Comparison of a Self-Directed and Therapist-Assisted
Telehealth Parent-Mediated Intervention for Children with
ASD: A Pilot
RCTAbstractIntroductionMethodParticipantsDesign and
ProcedureEligibility and Sample Characteristics MeasuresParent
Outcomes MeasuresParent Intervention FidelityParent Sense of
Competence Scale (PSOC)Family Impact Questionnaire

(FIQ)Child Outcome MeasuresLanguage TargetsMacArthur-
Bates Communicative Development Inventory (MCDI)Vineland
Adaptive Behavior Scales, Second Edition (VABS-II)Group
AssignmentSelf-Directed GroupTherapist-Assisted
GroupResultsAnalytic StrategyParent OutcomesChild
OutcomesDiscussionAcknowledgmentsReferences
Psychological Services
Customizing a Clinical App to Reduce Hazardous
Drinking Among Veterans in Primary Care
Daniel M. Blonigen, Brooke Harris-Olenak, Jon Randolph
Haber, Eric Kuhn, Christine Timko, Keith
Humphreys, and Patrick L. Dulin
Online First Publication, November 8, 2018.
http://dx.doi.org/10.1037/ser0000300
CITATION
Blonigen, D. M., Harris-Olenak, B., Haber, J. R., Kuhn, E.,
Timko, C., Humphreys, K., & Dulin, P. L.
(2018, November 8). Customizing a Clinical App to Reduce
Hazardous Drinking Among Veterans
in Primary Care. Psychological Services. Advance online
publication.
http://dx.doi.org/10.1037/ser0000300
Customizing a Clinical App to Reduce Hazardous Drinking
Among
Veterans in Primary Care
Daniel M. Blonigen
Department of Veterans Affairs Palo Alto Health Care System,

Menlo Park, California, and Stanford University School of
Medicine
Brooke Harris-Olenak and Jon Randolph Haber
Menlo Park, California
Eric Kuhn, Christine Timko, and Keith Humphreys
Menlo Park, California, and Stanford University School of
Medicine
Patrick L. Dulin
University of Alaska–Anchorage
Within the Veterans Health Administration (VHA), 15–30% of
patients seen in primary care are identified as
hazardous drinkers, yet the vast majority of these patients
receive no intervention. Time constraints on
providers and patient-level barriers to in-person treatment
contribute to this problem. The scientific literature
provides a compelling case that mobile-based interventions can
reduce hazardous drinking and underscores
the role of peer support in behavioral change. Here, we describe
the benefits of using a clinical app–Step
Away–to treat hazardous drinking among VHA primary care
patients as well as an approach to customizing
the app to maximize its engagement and effectiveness with this
population. We highlight the value of
integrating use of Step Away with telephone support from a
trained VHA peer support specialist. This type
of integrated approach may provide the key therapeutic
components necessary to generate an effective and
easily implemented alcohol use intervention that can be made

available to VHA primary care patients who
screen positive for hazardous drinking but are unwilling or
unable to attend in-person treatment.
Keywords: Veterans Health Administration, primary care,
hazardous drinking, smartphone application,
peer support
Hazardous drinking is defined as a pattern of alcohol
consumption
that places an individual at risk for adverse health
consequences, even
though the individual may not meet criteria for a diagnosis of
alcohol
use disorder. Globally, hazardous drinking negatively impacts
medi-
cal treatment, increases the likelihood of chronic medical
conditions,
and increases health care utilization and costs (Whiteford et al.,
2013).
Among U.S. military veterans, alcohol is the most widely used
psy-
choactive substance and carries the greatest clinical burden on
the
Veterans Health Administration (VHA; Fuehrlein et al., 2016).
Im-
portantly, within the VHA, 15–30% of veterans who are seen in
primary care are identified as hazardous drinkers, yet the vast
majority
of these patients go untreated (Bradley et al., 2017).
Daniel M. Blonigen, Health Services Research and Development
Center for Innovation to Implementation, Department of
Veterans Af-
fairs Palo Alto Health Care System, Menlo Park, California, and

Clinical simulation evaluates health IT safety

Clinical simulation evaluates health IT safety

Recommended

Recommended

More Related Content

Similar to Clinical simulation evaluates health IT safety

Similar to Clinical simulation evaluates health IT safety (20)

More from bartholomeocoombs

More from bartholomeocoombs (20)

Recently uploaded

Recently uploaded (20)

Clinical simulation evaluates health IT safety