Ehr Testing Challenge

THE EHR TESTING CHALLENGE
Perfecting the Process to Yield Better Results

www.qasymphony.com • 1-844-798-4386
Contents
Overview............................................................................................................... 1
The Unique Challenges Healthcare Organizations Face........................... 2
Defining Testing Coverage, Scope & Priority Using Risk Analysis........... 3
The Must-have Testing Strategy...................................................................... 5
Selecting a Testing Method.............................................................................. 6
Workflow Distinction & Definition.................................................................. 9
Test Results & Defect Reporting..................................................................... 10
Conclusion............................................................................................................ 12

Electronic Health Records (EHR) have become an essential
part of the way healthcare organizations operate. But, the
quality of the EHR software has become a sore spot for
users. Because these systems are often highly complex,
interoperable and not-so-intuitive, testing by the end user
is absolutely critical. Why? Think of all the workflows in a
healthcare operation: from admissions, patient visits and
testing to labs, medications and prescriptions — not to
mention documentation requirements, interoperability
verification and analytics / reporting. Workflows also
vary amongst facilities, clinicians, physicians, accounting
and regulatory administrators. The possibilities are not
endless, but they are daunting.
Ultimately, healthcare systems themselves — be it a
physician practice, practice group, acute hospital system
or a post-acute system — are responsible for patient
safety, dosage accuracy, data integrity and patient
information security. And yet, healthcare systems
generally don’t have teams of professional software testers
or quality analysts that test their software and ensure that
everything is running smoothly — for all users. That testing
burden often falls on the clinician, which means that
it’s often the last step and one that’s performed under
extreme time pressure. For that reason, it’s rare that
these healthcare systems are able to implement testing
processes that scale and allow for re-usability thanks to
the great pressure surrounding testing.
A common complaint amongst EHR users is that they
don’t know what exactly they need to test. They make
statements like: “As an end user, I don’t access the back-
end systems to view code or process details so I’m reliant
on being able to see, touch and test all the UI workflows
for each user type.”
What’s more, interoperability testing is a difficult task since
those systems are designed to run in production, not in
test. Most can be configured to test on a staging server,
but the environment is never exactly the same as production.
With the Right Tools and Processes in Place, EHR Testing Can
Actually be Incredibly Effective and Perfectly Painless.
Overview

Additionally, there are multiple types of interoperability systems live at
any given time — and typically from multiple vendors. In most systems,
there’s pharmacy (prescriptions), patient-facing documents and records,
lab test results, and MRI or other imaging-related records — not to
mention medical devices and software-controlled surgery devices.
With each additional system comes more test formats, test cases and
repositories, and less traceability back to what change in which system
caused a test to fail.
All this proves that the depth and breadth of tests that must be
performed each and every time software is updated can be a monster
to manage. So how can healthcare systems wrangle it in the midst of
all their time and resource constraints? Even with the perfect testing
strategy and tools, it isn’t easy. End user testing is critical to both patient
care and clinician morale, and a poorly tested EHR negatively impacts the
health of the business itself.
If we agree user testing is essential, then what can be done to create
an effective and efficient method of testing? Ultimately, just like patient
intake or discharge, software testing is a critical workflow for hospitals
that requires attention all its own. By investing time in the development
and optimization of a sound testing strategy, healthcare organizations
can ensure the quality of their applications, and use the software to their
best advantage.
In this white paper, we offer healthcare organizations insight into techniques
that will help them continuously improve their testing practices.
The Unique Challenges Healthcare Organizations Face
Testing EHR software for a healthcare organization presents unique
challenges. How do you know what to test? What are all the “things”
you should test? The objective, or point, of testing is to ensure the
application meets your regulatory needs, your workflow needs, and your
users trust it to provide valid and accurate information. The first step
in ensuring testing success is planning a test strategy. A test strategy
defines testing scope, priority based on analyzing the risk factors
for each application workflow or operation, based both on past
history as well as recent configurations and changes.
DOCUMENTS &
RECORDS
The many systems involved
in patient care can make it
incredibly difficult to determine
what causes test failures
DOCUMENTS &
RECORDS
MRI OR OTHER
IMAGE RECORDS
LAB TEST
RESULTS
PHAMARCY

End users don’t need to be professional software testers if
they have a clear plan and set of testing assets to execute.
When you have a plan that defines what’s tested, by whom
and how deep into each workflow the tester needs to go,
end users are empowered to test applications the right way.
Developing a plan is as easy as:
• Defining end user group workflows
• Mapping out clinician workflows for each non-
physician role
• Mapping out physician workflows — and don’t forget
lab results, radiology, pharmacy and reports, both
patient-and system-based.
• Using risk analysis to define your test coverage
Defining Testing Coverage, Scope &
Priority Using Risk Analysis
Risk analysis is a method of determining what needs to be
tested for each workflow and ranking it by priority. As end
users, you know which workflows are critical and which
are lower priority. Keep in mind that testing occurs on all
priority levels so don’t think that only the critical workflows
are tested. Risk analysis provides a way to narrow down
which workflows and functions should be tested first and
most frequently. Some workflows need to be tested with
multiple data points and across different environments,
while others might be less mission-critical and require one
pass — or may not need to be tested at all.
Risk analysis can be done using a spreadsheet, a Word
document or — better yet — a purpose-built test case
management platform that can allow for dynamic planning
and prioritization. For simplicity’s sake, the example we
describe here uses a spreadsheet for risk analysis.
When creating a risk analysis spreadsheet, the left side
should list the workflows and their functions — such as
all the tasks a clinician performs in a software application.
It would also include an admission workflow, including
the tasks for admitting patients, capturing accurate
demographic data, allergy information, medication and
health history. Additionally, users may be responsible
for documenting and verifying insurance coverage by
payer. We’ve barely scratched the surface in the patient
admission process, and already there’s a large amount of
testing to cover.
In addition to helping simplify and document the workflow
processes, the risk analysis grid offers additional distinct
advantages. Once you’ve done the initial work of creating
it, the grid is easy to update and use the next time you
need to plan testing.
Risk analysis is a method of determining
what needs to be tested for each
workflow and ranking it by priority.

Given typical unforeseen hiccups, we recognize we’re never able to test every workflow we want to in the depth we’d like,
so establishing priorities up front allows us to focus on core testing responsibilities as timelines shorten.
Too often, healthcare organizations only think about testing when their testing process is already in flight, and they
don’t spend the necessary time between test cycles to retrospectively optimize the process. With the approach we’ve
described here, you’ll know — and be able to prove — that your software works accurately for every functional use
within the organization.
The risk assessment score is determined by multiplying the
assigned weight (impact) by the business risk (likelihood
percentage). So, the most risky areas are the ones where the
potential impact is high (i.e. patient death) and the likelihood is
also high (i.e. part of the new patient enrollment path).
If you look at the example “grid” above, you’ll see the functions listed on the left as well as the various “weight” factors
that create a calculation and determine the risk value on the right. For a healthcare organization, you’ll want to change the
headings to match your needs, and as a team determine the rankings of each defined function. Once each is ranked, then
the final calculation is used to determine what value constitutes a high, medium or low risk.
Once the risk level is determined, create your test strategy by ranking the functions in priority order based on risk. When
pressed for time, start with the highs, mediums and then lows. In subsequent testing rounds, consider mixing the risks
and testing the mix to ensure full coverage without testing every function every time.
Please note that risk scores will vary in each particular organization.
HIGH
Risk score
> 467
MEDIUM
Risk score
>234 and <466
LOW
Risk score
zero and < 233
DEFECT/FUNCTION/ACCEPTANCE CRITERIA
ASSIGNED
WEIGHT
BUSINESS RISK
FUNCTIONAL
IMPORTANCE
...
RISK
SCORE
RISK LEVEL
WEIGHT SCORE WEIGHT SCORE ...
Medication Orders FDB 10 10 100 10 100 ... 600 High
Medication Orders Direct 5 10 50 5 50 ... 190 Low
Medication Orders Search 7 10 70 7 49 ... 329 Medium
Medication Orders Copy 5 5 25 5 25 ... 175 Medium
Order Sets FDB 10 10 100 10 100 ... 640 Low

TEST STRATEGY TEMPLATE
<Project or Release>
<Author name - Version 1.0>
Project definition & objective
<A brief introduction of the overall project.>
Test Definition
Features Tested
<List all the application functions that are
planned for testing.>
Test Deliverables
Testing Tasks
Develop test cases.
Conduct test cases reviews with team
members.
Perform tests.
Report bugs.
Update the Team’s Risk Analysis grid
(if needed).
The Must-have Testing Strategy
In software development teams, the testing strategy is generally called a “test plan.” Traditionally it’s a long document
that’s meant to be reviewed and approved by upper management prior to any test execution. In reality, the test plan is an
extremely long and tedious read, containing the detail of all test cases and requirements that nearly no one, except the
authoring tester, reads.
The testing strategy is meant to be short, concise and to the point so users at any level can read through it in under 10
minutes. It’s an outline that includes a description of who’s testing what, where and when, and it provides the organization
with a history of the testing effort as well as an overall plan. Again, it’s short, quick and concise — just like the generalized
template below.
TEST CASE TEMPLATE
<List the test case ID and title or brief
description OR include the path to locate >
ID# –Meds Summary Window_Physician
Testing Resources & Responsibility
<List the testing resources assigned and
their QA responsibilities.>
Testing Environment, Date/Time
<List the testing resources assigned and
their QA responsibilities.>
Risk & Mitigation
<List any known risks and their mitigation
strategy.>
RISK MITIGATION & COMMENTS
<Example>
Full regression testing that includes different client types is not possible Selected 2 standard customer scenarios (Medicare A, Medicaid).
1
2
3
1
2
3

If you search the Internet, you’ll find a number of different free
templates and samples for testing strategies. The important idea is
to find the one that meets your needs and will be adopted by your
testers. The testing strategy should be a living, breathing document
that’s read and understood by all parties involved in the testing effort.
Minimally speaking, the testing strategy defines who’s testing what,
where and when. Once execution is complete, it can also be used to
track results if you don’t want to use a separate document. Sometimes
minimizing the number of documents that have to be created, tracked
and used results in greater team efficiency.
Once a testing strategy is defined, it’s subject to change until testing
is complete since you want the testing strategy to be a record of the
testing event so it serves as a historical record. By keeping a record
of each testing effort, the team can learn from the past and make
the process better by implementing changes to it. Continuously
improving testing is only possible if past testing events are
documented or known.
Additionally, the testing strategy is a useful tool for communicating
with upper management or even vendors who need to know what
tests are being planned and executed. This keeps the organization’s
resources from having to repeat what’s being tested, where, when
and by whom to multiple sources. They can simply share the testing
strategy and focus on actual work to be done.
Selecting a Testing Method
Once you’ve gotten the risks documented and measured and the
organization’s testing strategy is defined, you’re ready to select a
testing method. But how do you know what testing method will work
best for your organization, or which one will provide the best results
Let’s break it down.
It’s hard to sift through all the buzzwords in the software development
industry — one of the most popular of which is Agile. Agile is a great
methodology for teams with dedicated, experienced developers

7 www.qasymphony.com • 1-844-798-43867
and testers who work closely together and collaborate
frequently. But for healthcare organizations, Agile has
significant shortcomings in terms of adoption. Agile
is meant to be fast and flexible, which doesn’t always
translate well in the highly regulated healthcare industry.
In addition, Agile doesn’t always work well when many
systems need to be coordinated on a single release cycle
— which is often the case in healthcare.
Another problem with Agile for healthcare application
testing is that at the end user level, there really isn’t room
for flexibility. Either the system works or it doesn’t — you
can’t be flexible on regulations, dosage accuracy or vitals
monitoring. Close is not good enough when it comes
to patient care or proving that an organization meets
government mandatory regulations.
One popular technique for overcoming this is building a
more traditional — or waterfall — suite of manual test
cases managed in a dedicated test case management
system. Implementing test case management to house
your suite takes time and effort, but it is a long-term win.
The tests are re-usable for as long as the software is in
place, and results can be tracked release over release.
While the tests may need to be updated if workflows or
major software changes occur, they can generally be used
for many years without major effort. For most test case
management tools, manual test case suites are defined as
scripted tests that are written as step-by-step instructions
on how to proceed through a function or workflow and
provide documentation on the expected results.
If you want to give testers greater freedom to challenge
your system in new and exciting ways, you should try
exploratory testing, which gives your testers a more
flexible approach. Once testers are satisfied with the
expected workflow processes, exploratory testing
empowers them to use your system like real patients and
clinicians would — and document new errors while in the
actual experience. None of the barriers that traditionally
come with a scripted test case are there to get in the
way. Instead, a tester can take many different paths in
an attempt to “break” the application. For example, they
can try to get the system to generate an incorrect dosage
calculation, place a medication order on a patient with a
severe allergy to it, pull a document onto the wrong user
record or attach a prescription for patient A on patient B.

With exploratory testing, anything a user can think of that
could make an application fail is fair game.
One important thing to remember when using exploratory
testing is to take clear notes on what’s been tested to
ensure adequate coverage in case of an audit. Many
vendors will provide automated documentation tools to
assist in this effort since clinicians may not have the time
or knowledge to capture this documentation during the
actual testing cycle. It’s also essential that testers focus on
finding errors and not following “rules.” Exploratory testing
is meant to find errors outside the lines, and it adds value
to testing because it finds errors that may go unnoticed
until a user makes an entry mistake or skips a step or two
in the expected workflow.
If an organization is pressed for time and the testing
resources don’t need detailed or explicit test steps,
consider using functional checklists. Checklists are simply
lists of the main functions each user role performs and
the expected result. Using your risk analysis grid data,
just create simple checklists using the application of your
choice with check boxes to indicate they’re complete.
Checklists work well when the resources doing the testing
are intimately familiar with the software application and
the role workflows. Also, checklists are relatively easy to
save, update and manage online or offline. Just remember:
their success at testing depends on the knowledge of the
tester as well as the quality of the testing process and the
assets provided.
Lastly, another process popular with many vendors
is automated testing. Automated testing works best
for organizations with more extensive IT or software
development resources. Automated testing is great for
performing repetitive tasks and validating scenarios with
multiple rows of data, but it doesn’t replace the need for
testers to define the automated tests needed and validate
the end user workflow experience. Automation also tends
to require regular maintenance, so increasing the number
of automated tests will require an increase in the IT
personnel necessary to support them.
What tools are available to give control and visibility into
all of these different types of testing present in today’s
leading hospitals? Using spreadsheets is common in the
healthcare arena — but spreadsheets are limited
in functionality.
If you’re using exploratory testing, it’s
important to remember to take clear notes
on what’s been tested so you have adequate
coverage in case of an audit.

9www.qasymphony.com • 1-844-798-43869
Spreadsheets don’t provide useful test metric information and
they’re difficult to manage efficiently, especially in large teams. Just
as hospitals invested in EHR systems as their patient data and
workflows grew in complexity, they’re now investing in test case
management solutions to replace their growing repository of test
cases managed in spreadsheets.
Workflow Distinction & Definition
When defining role-based tests, remember to include all the roles
affected by the software. Plan who’s going to execute each workflow
— from clinicians and physicians to administrators — and make sure
any changes made up front don’t impact downstream accounting or
finance functions. It’s especially important when there are questions
about functionality or the user role isn’t actually the person doing
the testing. For example, it’s rare for physicians, administration
heads or the CFO to actually test the software, and if they aren’t
going to test their workflows, it’s critical that whoever does has a
well-defined test script or functional checklist that incorporates
those users’ needs and expectations.
When creating workflows, it’s important to interview or collect input
for each role. You’ll want to find out:
• What frustrates them in the software?
• Where exactly does the software fail them?
• Do they have to duplicate tasks?
• Can they get to the information they need quickly?
• What functions in the software do they not trust?
Often, discussions about how to best test software become
insightful conversations about how to best build and configure the
application. One common EHR complaint amongst physicians, for
example, is that software adds unnecessary tasks that interrupt
their workflow and causes them rework. Similarly, clinicians often
complain about inaccurate lab results with incorrect or missing value
indicators. Knowing what parts of the software are problematic to
end users helps you plan the testing effort and understand where to
focus the most attention.
A common EHR
complaint amongst
physicians is that
software adds
unnecessary tasks
that interrupt their
workflow and causes
rework.

Once a testing method is chosen and the test strategy and role workflows are defined, it’s time verify. When using role
defined workflows, the testing results are highly dependent on the validity of the workflow, so make sure at least one expert
from each area reviews the workflow tests fully. A solid, reviewed test case provides improved and valid testing results.
Test Results & Defect Reporting
As the testing effort progresses, defects will be found and documented, and will likely need to be reported. When
reporting these issues to vendors or support personnel, it’s critical to explain the steps that led to the failure in detail so it
can be replicated.
First, capture the title and description in a format that includes the functional area, followed by the role or workflow and
then a short description. For example, as a clinician discovers that when he sends a verbal medication order to a physician
for approval, the application fails to update the medication order status during the active session. He has to save, log
out and log back in to see the correct status. So how can the clinician explain these errors to the vendor in a way that
allows them to understand the nature of the problem and the need for an immediate fix? The statement below is a good
example of how to communicate the issue:
Medication processing physician approval request status fails to
update during active session.
In this statement, the functional area is first, so it’s easy to see where the problem occurs; next comes the role it affects,
accompanied by a succinct description. Last is the status, worded in a way that that’s clear and concise, and tells software
development team exactly what’s wrong. Next, follow up with accurate steps to reproduce.
When reporting
failures to vendors
or support
personnel, it’s
critical to explain
the steps you took
in detail.

The steps to reproduce the issue are key for error
reproduction and getting the issue corrected as soon
as possible. Be as detailed and explicit as you can, and
remember that you’re explaining your steps to a
complete stranger with the end goal of getting the
problem corrected. Below is a good example.
Example:
“Log in” as a clinician with access to
place medication orders.
Navigate to the physician order entry
page by clicking the “Medication
Orders” link.
Select a patient with multiple existing
and approved medication orders.
“Add” an order for Levothyroxine
125 mcg with a frequency of Daily
for 90 days.
Click the option to “Send to Physician”
for approval button.
Click “Refresh” or allow the window to
refresh and updated the view.
View the order details and the
order status.
Clear, concise and detailed. It’s important to indicate the
user role that’s logged in and performing the action.
It’s also significant that the problem occurs on patients
with existing orders. These steps are configuration or
situational indicators that are helpful for development
teams when trying to reproduce a customer user defect.
In step 5, you’re indicating the action you’re trying to take,
but you also want to include what you expect to occur and
what’s actually happening.
After the steps to reproduce, add two additional sections:
“Expected Results” and “Actual Results”. For the example
above, the expected and actual results look like:
Expected Results: The “Send to Physician for Approval”
button should update the medication order status to “Sent
for Approval” immediately.
Actual Result: The “Send to Physician for Approval”
button remains as “Draft Order”. The status fails to
update unless the user saves, logs out of the application
and then logs back in. Once the user logs in again, the
status is updated.
Since EHR applications tend to be highly configurable,
fully describing the problem and what the user expects
to see is essential for communicating the nature of a
problem and its impact on the end user. Clinicians often
struggle with getting the detailed reproduction data they
need to close out defects on the first attempt, which is
where a defect documentation tool like qTest eXplorer
becomes invaluable. These tools record every aspect
of the testing session and automatically create detailed
documentation. Defect reports written with distinct
steps, expected results and a problem description are
more likely to get fixed — and the clearer the problem is,
the less likely it is to get pushed off to later or ignored due to
a lack of understanding. Using a tool like qTest eXplorer also
helps with documentation for an audit — making the entire
process easier and more effective.
1
2
3
4
5
6
7

12
Conclusion
EHR software updates occur on a regular basis — and each time they
do, it creates risk. The challenges of testing an EHR application for each
update across all user roles and functions can only be overcome with
a good testing strategy, a full understanding of the risk areas, clearly
defined user workflows and testing methods that are maintainable and
re-usable.
When testing, it’s critical to maintain a history of the effort so future
events can be improved as needed. Knowing that testing will result
in finding defects, reporting them clearly and concisely yields a plan
for working around errors and getting issues fixed faster. Too many
healthcare organizations don’t have the data they need to make
accurate testing decisions for one simple reason: they’re struggling to
manage their testing activities in spreadsheets.
EHR software is as complex as healthcare organizations themselves,
and the variations in workflow and configuration create systems prone
to failure. The burden of ensuring that systems work, falls on the end
user at the point of patient care — making end user testing essential to
the overall success of the organization, its employees and its patients.
For all these reasons, organized end user testing is not optional for
healthcare organizations; it can be the difference between success and
failure. When it’s planned, organized and executed with methods suited
to the organization, testing can be effectively and efficiently executed —
without all the aches and pains.
Protect your business, employees and patients — test often and test
organized for far better results.
References
• Becker’s Health IT & CIO Review, The problem with EHRs: 5 complaints from CIOs,
Akanksha Jayanthi, January 20, 2015.
• InPracSys, TOP 7 PHYSICIAN EHR COMPLAINTS, Vlad Hurduc.
• Testing Computer Software, Kaner, Nguyen, Falk. 1999 Wiley Publishing.
• How to Break Software, James A Whittaker, 2003 Addison-Wesley
Protect your
business,
employees and
patients — test
often and test
organized for far
better results.

QASymphony was named a Cool Vendor in Application Development by
Gartner in 2015 and is headquartered in Atlanta, GA.
Learn more at www.QASymphony.com or call 844-798-4386
To create solutions for Agile development
teams that significantly improve speed,
efficiency and collaboration throughout
the software testing process.
THE QASYMPHONY MISSION
—
Sign-up for a FREE TRIAL | Request a FREE PERSONAL DEMO

Ehr Testing Challenge

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to Ehr Testing Challenge

Similar to Ehr Testing Challenge (20)

More from QASymphony

More from QASymphony (20)

Recently uploaded

Recently uploaded (20)

Ehr Testing Challenge