The WALEX process combines facilitated discussion and discrete choice experiments to define the attributes of a realistic point-of-care test for Chlamydia trachomatis. Key findings include: 1) Participants were willing to accept longer test times and slightly higher instrument costs to preserve attributes like lower test cost, vaginal sample type, and processing multiple samples per batch. 2) Analysis of survey data identified where participants were willing to compromise on attributes and where they were unwilling. 3) The methodology provides a way to understand user needs and tradeoffs to help bridge the gap between need and realization of new diagnostic technologies.

1. Table 4: Question Pairs Applied to
Forced Choice Surveys
Participants must choose from the best
choice even if it is not the ideal choice
Driving Point of Care Technology Development:
An integrated approach to consensus building using DCE and facilitated meetings
Joany Jackman*, Mary Jett-Goheen, Terry Hogan, Yu-Hsiang Hsieh, Anne Rompalo and Charlotte A. Gaydos
Johns Hopkins University Applied Physics Laboratory* and Johns Hopkins Medical Institutes
Abstract:
Background: The need for a Point of Care Test (POCT) for the diagnoses
of Chlamydia trachomatis (Ct) is highly significant. These infections are
particularly hazardous to women since many are asymptomatic. Despite
the recognition that POCT for Ct is an urgent public health need and that
technology advances in diagnostics have made rapid POCT realistic, no
reliable POCT for Ct exists.
Objective: Our studies describe a methodology which can be used to
address the barriers between the need for a Ct POCT and its realization.
Methodology: We employed a process known as WALEX (Warfare
Analysis Laboratory Exercise) in combination with Design of Experiment
(DOE) methodology using discrete choice experiments (DCE), to define
the attributes of the most realistic (as opposed to the ideal) POCT. The
WALEX is an interactive oral and simultaneous electronic discussion
between experts in different fields linked by a common interest in
development of Ct POCT. The use of DOE methods in creation of DCE
provided defined metrics by which trade-offs could be measured.
Results: Our studies highlighted features of an ideal Ct POCT, ranked
those most critical to test acceptance and those open to negotiation.
End users were more flexible on the time required to conduct an ideal
test and one time instrument costs, if the requirement for higher assay
throughput, lowest test cost and source collection (vaginal swabs) were
preserved as necessary assay characteristics.
Conclusion: WALEX in combination with DCE helped us identify the gap
between perceived and actual requirement for acceptable Ct POCT,
addressed the most realistic commercial solutions to bridge those gaps
and provided metrics regarding tradeoffs which end users were willing
to make.
Implications: Multidisciplinary approaches which involve multiple
disciplines (end user, developer and regulatory) provide better
understanding of the real versus perceived needs for Ct POCT.
Figure 1: WALEX process
Facility
Design
Software
support
Analytical Planning
Components
Warfare Analysis Laboratory Exercise shown above
describes a process developed for the DOD for war gaming
specific problems. The WALEX meeting is different from a
focus group in that the group participants in oral
discussion and an electronic discussion format
simultaneously. This gives all an opportunity to speak and
comment on issues in both formats.
We applied this process to address the gap between the
need for Chlamydia POCT and the development of
A. Demographic Breakdown of WALEX participants and Survey
Participants by Profession
WALEX Survey
N (%) N ( % )
Research Scientists: 7 (26%) 7 (31%)
Medical Doctors: 5 (18%) 5 (23%)
Gov. Sponsor/PM: 9 (33%) 4*(18%)
Other: 6 (22%) 6 (27%)
Total: 27 22
*5 US Government personnel did not participate the surveys due to work related restrictions
B. Self Reported Expertise of Survey
Participants in Chlamydia or POCT
Category Frequency (%)
Chlamydia Researcher 13
Chlamydia Clinician 9
Involved in POCT development 52
Familiar with diagnostic
testing requirements 26
Table 1A-B: WALEX Participants
Choice Set
Number
Participant
Choice
Frequency
(%)
Sample Type
Hands-on
Time
Instrument
Time
Samples
per Batch
Instrument
Set-up Cost
Assay Cost
1 91 Cervical/Urethr
al Swab
5 min 20 min 1 $20000 $10.00
1 9 Cervical/Urethr
al Swab
10 min 5 min 1 $5000 $50.00
2 18 Cervical/Urethr
al Swab
10 min 20 min 4 $5000 $25.00
2 82 Cervical/Urethr
al Swab
10 min 4 hr 1 $20000 $10.00
3 32 Vaginal Swab 5 min 5 min 1 $5000 $50.00
3 68 Cervical/Urethr
al Swab
10 min 5 min 4 $5000 $50.00
4 48 Cervical/Urethr
al Swab
5 min 4 hr 1 $10000 $25.00
4 52 Urine 10min 1 hr 1 $10000 $10.00
5 64 Cervical/Urethr
al Swab
10 min 5 min 2 $5000 $10.00
5 36 Vaginal Swab 10 min 5 min 4 $20000 $50.00
6 95 Urine 10 min 5 min 1 $5000 $25.00
6 5 Cervical/Urethr
al Swab
5 min 1 hr 1 $10000 $25.00
7 95 Cervical/Urethr
al Swab
10 min 5 min 2 $10000 $25.00
7 5 Urine 5 min 5 min 4 $5000 $25.00
8 14 Cervical/Urethr
al Swab
5 min 5 min 4 $5000 $10.00
8 86 Cervical/Urethr
al Swab
10 min 20 min 2 $10000 $10.00
9 62 Urine 10 min 5 min 2 $20000 $10.00
9 38 Urine 5 min 5 min 1 $10000 $50.00
10 38 Cervical/Urethr
al Swab
10 min 4 hr 1 $10000 $50.00
10 62 Urine 5 min 4 hr 1 $20000 $25.00
11 29 Cervical/Urethr
al Swab
10 min 1 hr 2 $20000 $50.00
11 71 Vaginal Swab 10 min 4 hr 2 $5000 $10.00
12 73 Cervical/Urethr
al Swab
10 min 1 hr 4 $10000 $25.00
12 27 Vaginal Swab 10 20 min 2 $20000 $25.00
13 5 Vaginal Swab 5 1 hr 1 $20000 $10.00
13 95 Urine 10 1 hr 4 $20000 $50.00
14 41 Cervical/Urethr
al Swab
5 1 hr 1 $20000 $25.00
14 59 Urine 5 4 hr 2 $20000 $50.00
15 86 Urine 5 1 hr 2 $5000 $50.00
15 14 Urine 10 20 min 1 $5000 $50.00
16 14 Vaginal Swab 10 1 hr 1 $5000 $25.00
16 8 Vaginal Swab 5 20 min 2 $5000 $50.00
l----------------Attribute Selection-----------|
Topic Area: POCT
Technical Requirements
Variables discussed
Patient acceptance of
sampling location
Cervix, vaginal, penile, urethral and urine as sites
Male preference for urine
Patient willingness to self
sample
Acceptance of self collection of vaginal swabs by women 14
Acceptance and preference of urine (in a cup) by men
Fewer bathrooms for collection in clinics for urine
Home testing acceptance of urine
Sensitivity of sampling site Organism load of urine, cervical swabs.
Organism load associated with first urine versus entire urine stream
Glans (male) is an insensitive site to collect by swab
Sample processing Requirements for concentration of urine: more equipment, more steps, more
training
Use of self collection sponge or urine pellet on swab
Specificity and sensitivity FDA regulates test approval
FDA states new test must perform >95% sensitivity of current NAAT
Chlamydia test15
Use two test system to capture and immediately treat positives and confirm
negatives by more sensitive test
Asymptomatic nature of Chlamydia is problematic in physician choice for
testing and syndromic treatment model
Lower specificity tests may result in faster detection and treatment of positives
No false positives required for fast test for Chlamydia
False positive and negative rates are driven by prevalence of disease in
population
Different Chlamydia disease prevalence rates need different specificity and
sensitivities
Time to result Patients want result in 5 minutes but will accept 20 minutes
ED physicians will accept up to 4 hours to fit with clinic flow
Public health laboratories want POCT to fit with clinic visit flow (<1hr)
Many patients tested for Chlamydia do not return for results
Other Use of screening tests versus confirmatory testing
Unacceptablity of terms: screening and confirmatory tests since results are
used in the same way for treatment
Topic Area: POCT
Transition to Commercial
Project
Variables Discussed
CLIA/FDA approval Both FDA has oversight in categorization of CLIA tests
User community for CLIA waived and FDA approved tests have different
levels of implied expertise
Clearance/approval requires new test have 95% agreement with reference test
Low complexity tests are candidates for CLIA waived tests
Reference test can be selected from any clinically approve tests performed by
trained professional
Unit Size Packaging Devices need to run tests in parallel
Batch tests for POCT require high volume and may not be appropriate for all
settings
Flexibility in number of samples is needed to address variations in patient
flow
Quality controls If quality controls are built into every run method they must be run with each
batch as defined by manufacturer’s instructions
CLIA inspectors not users define “a run” or required periodicity of controls
Manufacture’s definitions of periodicity do not support clinical assay work
flow
Manufacturer should design controls and stability testing to support “lot level”
testing not “shipment level”.
Topic Area: Risk
Assessment and Cost
Variables Discussed
Cost structure Costs per test are dependent on volume of tests produced
Cost of POCT are dependent on whether another test is required to confirm
result
Costs of $30 are too high for screening test
If number of people were being tested that CDC recommends, volume of tests
would increase
Current NAATs test cost approximately $15
Multiplexed test Combining tests is a strategy for reducing cost
Multiplex tests are only good if co-prevalence rates are high
Multiplex tests costs which are three times the cost of a single assay are
unacceptable
Topic Area: Market
Acceptance
Variables Discussed
End users defined POCT end users defined as physicians, clinicians and patients
Clinical laboratorians defined “the lab” (themselves) as end users in order to
provide greatest cost control
CLIA waived tests would be appropriate
Internet can be used to market tests directly
Drivers for Chlamydia POCT POCT should be available as off the shelf product “at the supermarket”
Privacy is driver for Chlamydia tests
Package inserts describe the target for use of the test (who should be tested)
but do not describe where test should not be used.
Home testing Patient focus groups indicate they would accept test with high false positive
rates if they could perform themselves
Problems in home testing are seen when patients self medicate based on test
results
Home sampling combined with mailers for testing at central location would be
acceptable
Sample
Type:
Hands-
on
Time:
Instrumen
t Process
Time:
No.
Samples
per
Batch:
Instrumen
t Set-up
Cost:
Cost per
Assay:
•Vaginal
Swab
5 min 5 min 1 $ 5000 $10
•Urine 10 min 20 min 2 $10000 $25
•Cervical/
Urethral
Swab
1 hr
4hr
4 $20000 $50
Table 3: Key POCT Characteristics
and their attributes as identified by
STD focus groups.
WALEX Process Makes Use of Facilitated Discussion and Surveys
Table 5: Analysis by Firth Unbiased Estimator of the Overall
Choices:
Figure 2: Prediction Profiler of the STD survey with the
settings for the maximum utility
A. Independent attribute choices (no compromise) B. Interdependent attribute choices (with compromise)
Table 2 : Highlights From Each
Topic Area Discussed During
WALEX
<A key feature of the WALEX
meeting which differentiates it
from a focus group is that
participants are stakeholders in
development of use of POCT for
Chlamydia. The WALEX
intentionally mixes participants
from different backgrounds in
order to provide different
viewpoints of the same
problem. In this way, the
WALEX meeting offers
solutions. As shown in Table 1A
participants were derived from
both end users (physicians,
researchers, developers (other)
and regulatory agencies and
government sponsors. Table
1B describes the expertise of
the participants. As shown,
74% of the participants were
familiar with POCT or
Chlamydia testing.
< The WALEX oral and
electronic discussion format
covered four topic areas for
development of POCT
specifically for Chlamydia.
The total time devoted to the
WALEX was 4 hours. The topic
areas were:
•Technology and assay
requirements inclusive of the
POCT Characteristics listed in
Table 3, in addition to
performance characteristics of
a successful test.
•Barriers to or requirements
for successful transition of
tests to commercial market
including FDA/CLIA
requirements and assay
throughput
•Risks to development of tests
inclusive of unknown or gaps
regarding technology
solutions, sites of infection
and microbial co-infections
and price point considerations
for single versus multiple
POCT.
•Market Acceptance by end
users based on concept of
operations or test deployment
(Public health clinic vs.
Emergency Department), cost,
ease of use, patient
acceptance
5000
< The list of
characteristics of an
acceptable POCT and its
associated attributes were
derived from preferences
and needs suggested in
focus groups conducted
prior to the WALEX.
These were used to
develop the question sets
in the forced choice
experiments below.
Conclusions:
We have described a new methodology for evaluating diagnostic technology
requirements for Chlamydia POCT. This methodology integrates the use of force
choice surveys constructed and analyzed using design of experiments approaches
and the use of a facilitated meeting format compromising both oral and electronic
discussions.
Using this method we have shown where WALEX participants were willing to make
trade offs between attributes and where WALEX participants were unwilling to
compromise.
Our studies demonstrated that in order to preserve the attributes associated with
cost, sample type and number of samples per batch, participants were willing to
accept longer assay times and slightly higher one time instrument costs.
Acknowledgements: This work was supported by National Institutes of Health/National Institute of Biomedical Imaging
and Biotechnology Grant U54 EB007958. None of the authors have a conflict of interest relative to the publication.
< The list of
characteristics of an
acceptable POCT and its
associated attributes
were derived from
preferences and needs
suggested in focus
groups conducted prior
to the WALEX. These
were used to develop the
question sets in the
forced choice
experiments shown to
the left. The second
column show the
frequency of the choice
in each set . The
cumulative frequency for
each pair is 100%. Using
2 x 2 matrix factorial
design for all the
attribute levels listed in
Table 4, 648 test points
(23*22*24*23*23*23) could
be represented in as few
as 16 questions.
^Prediction Profiler is a JMP graphical representation of utility and allows one to look variation for each
attribute as an independent and interdependent variable. Analysis of survey data was evaluated based
on its desirability as independent and dependent conditions. Desirability as defined by JMP software is
a technique by which multiple attributes can be optimized to obtain the best characteristics for as many
attributes simultaneously. As independent variables, desirability for each individual attribute was
determined at maximum desirability (1.0). For interdependent variables, desirability is allowed to vary
to the optimal level for all factors even if desirability for a single factor is less than its optimal
independent value. A shown by comparison of graphs A and B, participants were unwilling to
compromise on the characteristics of sample type (vaginal) , number of samples per batch (4) or assay
costs ($10) but were willing to compromise slightly on
< Data were collected electronically and
analyzed using JMP version 8 (SAS Institute
Inc., Cary, NC) with application of Firth’s
unbiased estimator. The analysis was
conducted by applying Firth’s unbiased
estimator available in JMP software Effect of
Likelihood is a graphical tool for ranking
causes from most significant (Low
probability relative to Chi square value) to
least significant (High probability relative to
Chi square). In this graph, the choices are
shown on the y axis in order of probability
(highest to lowest). On the x axis are plotted
the L-R ChiSquare Values.