Usability Sample

BLOOD ANALYZER FINDINGS
November 2011

Client Name Product name

The goal of this project was to evaluate the
usability and desirability of a new piece of lab
equipment for emergency use.

This presentation was given to upper management
to focus future development of this product.

Client Name Product name 2

PROJECT OVERVIEW
KEY FINDINGS
DETAILED FINDINGS
SCANING
PACKAGING
TRANSFER/INNOCULATION
INSERTION & REMOVAL
PRINTING
RESULTS & FEEDBACK


PROJECT OVERVIEW
PROJECT GOALS
PARTICIPANT OVERVIEW
SESSION OVERVIEW


PROJECT OVERVIEW-GOALS
Evaluate responses and interactions of targeted
Point Of Care users in association with the
current PRODUCT; attention will be focused on
(but not limited to):
•Ease of use
•Ergonomics
•Fit with current workflow
•Evaluation of system workflow
•System size

Evaluate viability of the current system and
system interaction scheme. Specifically
packaging, card handling, inoculation, interface
between card and instrument, and removal of
the card.

While it was not in the original scope of the
project, we noted issues regarding the general
flow of the user interface within the context of
this study. This study was not intended to provide
feedback on the UI so findings regarding UI will
be kept at a very high-level. However, user
feedback suggests that the UI is not finalized and
will need further review.


PROJECT OVERVIEW- PARTICIPANT PROFILE
Institution
Had more than 250 beds and were considered
Participant overview infographic acute care hospitals with emergency
departments

Nurses
Worked in an Emergency Department in the last
two years.
Half had experience performing cardiac marker
testing themselves, while the other half did not
Nurses had a mix of experience level

Lab Techs
Had at least 3 years of experience working in a
core or stat lab

POC coordinators
Managed ED nurses or stat lab techs
Half work in hospitals with cardiac marker testing
in the ED

All Participants
Had a mix of age, gender, glove size, & hand
*Full profiles of each participant can be found in the appendix dominance


PROJECT OVERVIEW- SESSION OVERVIEW

Introduction Training Initial Impressions Trials
Participants discussed the Participants were shown a 3 Participants then discussed Participants completed a
types of tests that they are minute training video which their initial impressions of the number of different trials to
familiar with at point of care. covered basic workflow, concept based on the video. gain familiarity with the
inoculation techniques, and equipment. These trials helped
system features. This training participants articulate what
insured that all participants worked and didn’t work with
had the same base line the system. After the Chicago
knowledge of the process. sessions the trials were
changed slightly to reflect
product improvements made
between cities.


PROJECT OVERVIEW- SESSION OVERVIEW

Workflow Evaluation Preference Interview Feature Sort
Participants evaluated the Participants Participants were asked their Participants “summed up” each Participants
proposed workflow then evaluated preference with regards to the part of the system and sorted a list
created their own ideal each part of features that have multiple go- discussed on the changes they of features
workflow using stimuli cards. the process to-market options like transfer would make. from most
on a 7 point device, scanner, etc. important to
Lickertt scale. least
important.


HIGH-LEVEL FINDINGS
KEY TAKE AWAYS
INITIAL IMPRESSIONS
ERRORS
USER RATINGS
WORKFLOW CHALLENGES
VALUE PROPOSITION


HIGH LEVEL FINDINGS- KEY FINDINGS
We thought: Now we know:

Participants may be unhappy with the insertion and
Insertion and removal is acceptable to users.
removal process

Inoculation- specifically opening a vacutainer to
Participants would be comfortable with opening a use a transfer pipette is inherently “scary” to
vacutainer and using a transfer pipette nurses, however the Closed Tube Sampler (CTS)
was well received.

Nurses are currently comfortable conducting
some point of care tests bedside, and might
The PRODUCT would be accepted as a centralized system
expect to use the PRODUCT in the patient’s
room.

Participants who were not currently running
tests at bedside saw no problem with the size of
The size of the analyzer might be a concern for users.
the analyzer, and mentioned that the would find
space for it.


HIGH LEVEL FINDINGS- POSITIVE INITIAL IMPRESSIONS
Overall the PRODUCT was well
received and easy to use by all
participants. This graphic
shows the frequency of
positive concepts mentions
when participants were asked
their initial impressions of the
product.

The size of the words correlate
to the number of times each
concept was mentioned
positively.
Concept Mentions
easy to use 12
3 bays 11
time 11
barcode scanning 8
QC 8
CTS 7
durable 3
accuracy 2
Data management 2
whole blood 2
big screen 1
familiar (similar to istat) 1
flexibility in transfer options 1
high quality 1
print with out docking 1
uses cartridges 1


HIGH LEVEL FINDINGS- NEGATIVE INITIAL IMPRESSIONS
This graphic shows the
frequency of negative concepts
mentions when participants
were asked their initial
impressions of the product.

The size of the words correlate
to the number of times each
concept was mentioned
negatively.

Concept Mentions
not bedside 8
transfer method 7
size 4
overfill 3
time 3
availability 2
cost 1
might not work in a helicopter
or ambulance 1 • Initial impressions were generally more positive than negative
waste 1 • Negative mentions of “Size” and “Not Bedside” were more popular with current I-stat users
who were performing cardiac testing at bedside [ATLANTA]
• Transfer method (Transfer pipette and CTS) was the most frequently mentioned concept.
• Time was mentioned both positively and negatively. Currently I-Stat tests take 10 minutes but
lab results may take up to 90 minutes.


HIGH LEVEL FINDINGS- USER ERRORS
In total participants completed Percentage of Errors for user trials
180 trials, of those 40
encountered at least one error.
This number is relatively low Errors occurring in trials 1-3
and shows us that participants
were capable of easily learning 19%
the system.

Errors included:
Scan in wrong order
Slight under fill or overfill
Threw away pouch before scanning Errors occurring in trials 4-6*
Insert into wrong bay orientation
issues during insertion 5%
Touched lenses
Forgot to hit go
Open exterior box from top

Trials with no errors
76%

*Most of these errors were- participant threw
away package before scanning


This chart shows a breakdown Percentage of Errors for User Trials
of the 40 errors.
100%
90%
80%
70%
60%
50%
40%
30%
20%
8% 5% 5% 5%
10% 2% 1% 1% 1%
0%

*error only applicable to Chicago trials.
Barcode was added to the cartridge for **error only applicable to Atlanta trials. Interface was improved to
Atlanta direct users to specific bays for Atlanta


This chart shows a breakdown Percentage of Errors for User Trials
of the 40 errors.
100%
90%
80% Only Addressed by
70% happened adding a Happened infrequently and would
60% with transfer barcode to be alleviated with proper training
50% pipette the cartridge
40%
30%
20%
8% 5% 5% 5%
10% 2% 1% 1% 1%
0%



Of these errors, “scan in the Percentage of Errors for User Trials
wrong order” and “insert into
the wrong bay” are the most 100% Did not happen on This is a critical error
severe. 90% any of the first trials, that can be
80% meaning that overcome with the
70% confidence in using proper affordances.
60% the system lead to
50% this error
40%
30%
20%
8% 5% 5% 5%
10% 2% 1% 1% 1%
0%



HIGH LEVEL FINDINGS- USER RATINGS
When participants were asked
to rate the difficulty of each
part of the process, opening
the exterior box and pouch
rated the lowest.

These parts of the process are
not severe and would not
cause safety issues, but greatly
impact user experience.
Very Difficult

Very Easy
• USER ERRORS and factors affecting USER EXPERIENCE are both critical to understand when
making design changes.


HIGH LEVEL FINDINGS- USER RATINGS
We also asked participants to
what extent they agree or
disagree with the following Strongly Disagree Strongly Agree
statements. This graphic shows
and averaged rank of the
results.

• Focus on the design and look of the product, the user-interface, and the overall footprint of
the system.


HIGH LEVEL FINDINGS- WORKFLOW CHALLENGES

BEDSIDE VS.
SCANNING ORDER MULTIPLE CARTRIDGE
CENTRALIZED

1 2 3


HIGH LEVEL FINDINGS- CURRENT WORKFLOW
Below is the current product workflow:
Take patient blood
Press new test
Scan user ID
Scan patient/specimen ID
Scan cartridge or cartridge pouch
Open pouch
Inoculate
Insert cartridge into the analyzer


Bedside vs. Centralized
Some nurses mentioned that they would want analyzers in every
Nurses who are familiar with bedside testing would like to have
room.
most, or all of, the workflow happen at bedside.
While putting the current system on a cart may be a short term
The benefit of a bedside workflow:
• The nurse would not have to leave the patient during critical times solution, there are a lot of core challenges with equipment on
• The nurse could not possibly use the wrong patient’s blood to run a test carts.
• The nurse could teach the patient about the test, what the results
mean, and show that the hospital cares because they have the most Nurses seemed receptive to doing scanning and inoculation in the
advanced equipment patient room and moving to a centralized location to insert the
cartridge, but they would really prefer not to leave the room.

• Consider finding ways to make a bedside workflow possible


Scanning Order
The biggest challenge with the current workflow is scanning all of the items in the proper order. While all of the user groups expected to
scan 1. a user identifier, 2. a patient identifier, and 3. the cartridge/pouch, when they started moving fast they would scan in the wrong
order or skip a scan.

1

VS
2

3
• Consider creating a system that is order agnostic


Multiple Tests
We asked participants to take us through the workflow they would expect for running multiple tests on the same patient and while the
details varied there were two key themes:

Efficiency- some participants wanted to do this process as Safety- some participants wanted to go through the entire process
efficiently as possible. Scanning their id and the patient identifier including all scanning for each test. They saw this as the safest way
one time and opening, inoculating, scanning, and inserting all three to run multiple tests.
cartridges together.

• Offer hospitals the flexibility to dictate how a multi-cartridge workflow will work for
them


HIGH LEVEL FINDINGS- VALUE PROPOSITION
Participants were asked to sort
this list of attributes. This order
is an averaged score of all
users. The order was
consistent between user
groups and geographies.

Least Important Most Important
Supplier Cost of the Cost of the Technical Breadth of Amount of Advanced, Quality &
experience instrument reagents Support and automated time each highly Reliability of
with point of Service assays test takes automated the
care testing analyzers instruments
with lab-like & reagents
results


HIGH LEVEL FINDINGS- VALUE PROPOSITION
Participants talked candidly Easy to Manage and Maintain
about how a product like this To sell to this group talk about:
would be introduced to their •Quality control
hospital •Inventory control
•Speed
POC •Easy of use = lack of errors
Coordinators •Wave certification (if applicable)

Easy to Use Strong ROI
To sell to this group talk about: To sell to this group talk about:
•Better/faster patient results •Improved results lead to better
•Easy to use = more control patient outcomes and lower
over your patient’s care with Hospital risk
Nurses •Faster turn around time leads
out the hassle of dealing with
the lab
Board to higher patient satisfaction
•A modular system makes
expansions and replacements
faster and less expensive.

• In order to meet the needs of this diverse group, flexibility is key


DETAILED FINDINGS
SCANING
PACKAGING
TRANSFER/INNOCULATION
INSERTION & REMOVAL
PRINTING
RESULTS & FEEDBACK
USER INTERFACE


SCANING- FEEDBACK
“I like scanning the cartridge before I add the “The scanner I would prefer is…”
blood to it so scanning with the integrated
scanner works for me. Scanning with the
handheld scanner would not be my preference
but to me it was very easy. It seemed that one
time I had to move everything around. “
– Wayne

“It’s sturdier in the machine but I liked the
handheld thing because it picked up quick. My
nurses are the best in the world but they are
hard on things”
-Victoria

In general using the scanner was…
Very difficult Very easy


SCANING- TAKE-AWAYS
• Participants were comfortable using a scanner. • Some participants worried about touching the
Scanning is a familiar interaction, however there were external scanner for fear that it was dirty.
a number of errors caused by participants scanning • The cartridge pouch was difficult to scan using the
items in the incorrect order. integrated scanner, especially after the cartridge had
been removed.
• Some participants disposed of the cartridge pouch
before scanning during the inoculate first trial.

• The current integrated scanner is difficult to use • Nurses like the idea of moving part of the workflow
because it does not offer perceptual feedback to the patient’s bedside
• The green light is hard to see when using the • This is the current practice with i-stat
integrated scanner • Scanning the patient wristband (instead of the
specimen) reduces the possibility of
identification error
• Consider making the scanning workflow more
flexible
• Some hospitals will want the option to manually
enter patient ID for John Doe situations.


SCANING- HIGH LEVEL RECOMMENDATIONS

Short Term Mid Term Long Term
• Offer an external wired scanner • Adjust the internal scanner so •Develop a bedside workflow using
with a cradle as an add on users can gain visual feedback. wireless technology
option.
• Explore ways to make the
scanning UI more flexible.


EXTERIOR BOX- FEEDBACK
“I think it could be designed a little bit “The exterior box I would prefer is…”
differently. It wasn't hard, but it wasn't as simple
as it could be“
– Susan

"I could see nurses stocking the cartridges in
their IV tray"
-Brittany

"I am not a big fan of the size of the opening“
– Erin

Opening the Exterior Box was…


EXTERIOR BOX- TAKE-AWAYS
• Participants understood how to open the box, but • Some participants had a hard time ripping the
mentioned that they might want more flexibility. perforations
• Removing a cartridge pouch required 2 hands
• The current box does not facilitate a multiple
cartridge “Grab”

• Opening the box was clear. • Flexibility is key
• Some participants wanted a visual indicator of how • Inventory management is a game changing feature
many pouches remain in each box that is best supported by the T-slot box, however
• Some participants would still want to write the the T slot has the perception of being “too fussy” for
expiration date on the box, even if they are using the grab and dump ED
inventory management. • The product will not always be used with inventory
management or in high volumes making the t-slot a
hindrance for removing multiple cartridges at the
same time
• The wire rack is a good add on option


EXTERIOR BOX- HIGH LEVEL RECOMMENDATIONS

• Create a clearer/easier to pull tab •Create a multi-open box • Create a visual indicator for the
for the T-slot. number of cartridges remaining
• Use tighter perforation in the package
• Create a graphic area for users to
write the use-by date

• *Offer an improved version of
the wire rack system as an
optional feature/add-on


REMOVE & OPEN POUCH-FEEDBACK
“The first one I went to open I didn’t hit the little
notch as well. You have to yank it harder. It
would be nice if there was a little zip thing. I
could be simpler. It would be nice to have a little
arrow showing where the notch is. “
– Wayne

"once you get it open you still have to wrestle
it“
– Sandra

“It’s
a little tough.”
-Dana

"It could have a little bit more of a tear. If there
was room for it, just to kind of get you going a
little bit better. It was evident that you pull here
where the black dot is, and it pulled pretty easy
because I am not that strong. “- Crystal


REMOVE & OPEN POUCH-FEEDBACK
Removing the pouch was…

Opening the pouch was…

Removing the cartridge from the pouch was…


REMOVE & OPEN POUCH-TAKE-AWAYS
• There were no cognitive issues with removing or • A slightly larger T-slot may make removing pouches
opening the pouch. easier
• Tearing the pouch open was incredibly difficult
• Participants used the pouch to handle the cartridge
• Some participants expressed worries about the
desiccant pack.

• Removing the pouch is clear • Participants were all able to discern where and how
• There could be clearer indicators for where to open to open the pouch, they struggled with the physical
the pouch force required to open
• Participants may still hand write the date on the • Worried that opening the pouch too soon would
pouch and would like an area to do that. reduce the accuracy of the test.
• The prototype packages had 2 barcodes which was
confusing to some participants
• Some participants mentioned that the package
seemed a little bit bigger than it needs to be.


REMOVE & OPEN POUCH- HIGH LEVEL RECOMMENDATIONS

• Slightly larger slot on external • Easier to tear pouch with deeper • Automatic visual indicator on
box notches and a more forgiving pouch when time at room
• More obvious graphic cues for liner material. temperature has been exceeded.
where to open pouch
• Graphic area for marking
expiration date.


CARTRIDGE-FEEDBACK
"I like that you have the option to write in
patient ID, but you don't really have to"
-Sonia

"I wonder if there is an easier way, like a glucose
test where you just prick the finger.”
– Brittany


CARTRIDGE-TAKE-AWAYS
• Cartridge orientation was challenging for some • All of the participants used the handle correctly
participants. • Participants mentioned that having the barcode on
• Touch/no-touch areas were intuitive the cartridge itself was a great “fail-safe” in case they
threw the pouch away.
• Almost all of the participants held the cartridge using
their thumb when inserting
• Some noted that the bottom of the cartridge was
uncomfortable.

• Graphics on the cartridge did not help participants • Participants looked at the cartridge after the test for
determine how the cartridge should be oriented some indication that the test had been completed.
during transfer and insertion
• The sight port was well received as a concept but
challenging for participants to actually see because
the user’s hand blocks the line of sight.
• Participants liked the area to write patient
information.


CARTRIDGE-HIGH LEVEL RECOMMENDATIONS

• Add arrows to assist with • Consider ways to make the • Consider moving the sight port
orientation cartridge visually different after it slightly to make it more visible
• Add 3 blood drop graphic has been used. during inoculation.


TRANSFER-FEEDBACK
It is challenging until you get training. After I did “The transfer method I would prefer is…”
it once it was easy. There is something about
the word inoculate. I didn’t like it up there. I
knew what you meant, but there has to be a
better way to say it.
– Wayne

I really like the fact that you don’t have to pull
back and push. I like the fact that you just put it
up against and push down. But the other way I
just see biohazard everywhere and infection
Insert- very easier, in fact easier that i-stat
-Victoria

In general transferring blood was…


TRANSFER-TAKE-AWAYS
• The word ”inoculate“ was confusing or inappropriate • In a couple of cases the CTS broke or did not properly
for users release the vacuum
• There was a general fear surrounding opening • The blunt end of the CTS made it challenging to
vacutainers or handling used CTS devices, especially insert.
among nurses.

• Again the sight port was hard to see for some • Overcome the fear of transferring by offering a easy
participants because their hands or the vacutainer to use and safe closed tube sampler.
were in the way
• One participant suggested adding an icon to the
cartridge to show 3 blood drops (the amount she
thought was the appropriate fill).


TRANSFER-HIGH LEVEL RECOMMENDATIONS

Transfer

Short Term Mid Term
•On the interface change the word •Improve the CTS by making it:
inoculate to transfer • Safer
• A tighter fit with the cartridge
• Easier to use (requiring less
physical strength)


INSERT & REMOVE-FEEDBACK
“The very first time, I didn’t know if it had to go “The shutter I preferred “The shutter I preferred
at an angle because when the door was open it for Insertion was…” to Close was…”
looked like it more go at an angle. Once I knew
how to do it, it was extremely simple. But again
that goes in training. If they have been shown
how to do it, it becomes a 7 for me-” Wayne

“Initially you couldn't exactly see where it was
going, but once you got it, it was simple“
-Susan

Inserting the Cartridge was…

Removing the Cartridge was…


INSERT & REMOVE-TAKE-AWAYS
• There were a few small orientation errors with the • None of the participants fingers touched the sides or
first insertion. shutters when inserting and removing.
• Some participants tried to insert the cartridge above • Most users had to bend down to insert the first time
the handle for the shutter. but noted that they would be trained and not need
• In a few cases participants tried to insert the to bend down.
cartridge into a full slot
• The biggest cognitive challenge with insertion is
determining which port to use.

• The angle positioning of the ports makes them • Insertion and removal was physically easier than
difficult to see. Most users bent down for their first expected, but cognitively more challenging than
insertion but did not look at the ports again for expected. The active bay must be clearer.
subsequent insertions.


INSERT & REMOVE-HIGH LEVEL RECOMMENDATIONS

• Create affordances on the • Create a bay labeling system • Add design details that protrude
interface to make correct bay from the bay to show users
more obvious where the cartridge should go.
• Consider adding soft lighting
inside the bays


PRINTING-TAKE-AWAYS
• Participants expected to have some means of
printing, but did not intend on printing every result.
“The printer I would prefer is…”
• They would like to see 2 patient identifiers as well as
result ranges on the print out

• Participants did not notice that there was a printer
onboard
• They would rather have the print out come out of the
side or top of the unit

• Some shorter participants had to work to reach the
print out in the current location.

• Users will need the capability to print, but will
probably not print every result.
• Thermal paper is considered temporary and can
therefore not be a part of a patients chart, it must
be taped and photocopied.
• The current location of the integrated printer is
challenging. Consider offering an external printer
which “fits” with the system.


INVENTORY MANAGEMENT
• Most participants reacted very positively to the idea
of inventory management
• However, explaining what inventory management is,
and how it works was challenging.
• This could be a game changing feature, but it will
require a well thought out sales approach.
• Some participants worried that they would still need
a visual indicator on the package showing room
temperature expiration.


RESULTS & FEEDBACK
Results Feedback
• Currently the I-stat only lets the user see the • Participants were interested in visual
last result, so the ability to go back and see past differentiation of the interface for critical
results is a value add. values.
• Optional auditory feedback would be well
• This also means that there isn’t really a received for critical values and errors.
convention for dealing with previous results • Color uses- when possible follow human factors
(search by operator, search by patient, etc). standard regarding alert colors.

• Currently medical record systems in hospitals a
incredibly complicated ranging from paper
medical records to multiple OS’s per hospital.

• There was some interest in instantly updating
doctors with critical values, but feasibility may
be challenging right now.


USER INTERFACE-HIGH LEVEL TAKEAWAYS

NEW TEST SCANNING BAY ALIGNMENT


USER INTERFACE-NEW TEST
Difficult to see

Participants naturally touched
the green area to start a test.

• Remembering to press new test was consistently challenging for users.
• When creating workflows this was the one step that felt confusing or unnecessary.
• Consider adding a “next test” screen for users who are running multiple tests on the same
patient.


USER INTERFACE-SCANNING

• Participants were comfortable with the idea of scanning three items, but as they became
more confident with the system they were more likely to scan items out of order. There is no
clear mental model to help users remember the prescribed order of scanning.


USER INTERFACE- BAY ALIGNMENT

• Bay alignment was one of the most challenging parts of the interface. Brainstorm ways to
make bay alignment clearer and more connected to the physical analyzer.


SUMMARY
Insertion and removal is acceptable to users.

Inoculation- specifically opening a vacutainer to
use a transfer pipette is inherently “scary” to
nurses, however the Closed Tube Sampler (CTS)
was well received.

Nurses are currently comfortable conducting
some point of care tests bedside, and might
expect to use the PRODUCT in the patient’s
room.

Participants who were not currently running
tests at bedside saw no problem with the size
of the analyzer, and mentioned that the would
find space for it.


Usability Sample

Recommended

Recommended

More Related Content

Similar to Usability Sample

Similar to Usability Sample (20)

More from researchdriveninnovation

More from researchdriveninnovation (8)

Usability Sample