Presented at the 7th International Conference on Qualitative and Quantitative Methods in Libraries, Paris France. Application for the 'best fit' framework synthesis methodology to systematic review data extraction

1. Applying ‘best fit’
frameworks to
systematic review
data extraction
Andrea Miller-Nesbitt, Catherine Boden,
Andrew Booth, et al.
7t h International Conference on Qualitative and Quantitative
Methods in Libraries, Paris France
May 2015

2. Overview
• Background
• Systema2c
review
process
• ‘Best
fit’
framework
methodology
• Applica2on
of
methodology
to
our
project

3. Background
Conduct
a
systema2c
review
to
address
one
of
the
15
ques2ons
iden2fied
in
the
MLA
Research
Agenda:
Appraising
the
Best
Available
Evidence

4. Research question
What
skills
and
knowledge
must
librarians
possess
in
order
to
be
able
to
design
tools
to
help
researchers
visualize,
mine,
and
otherwise
manage
large
and
complex
data
gathered
during
both
quan2ta2ve
and
qualita2ve
research?

5. Research question
What
skills
and
knowledge
must
librarians
possess
in
order
to
be
able
to
design
tools
to
help
researchers
visualize,
mine,
and
otherwise
manage
large
and
complex
data
gathered
during
both
quan2ta2ve
and
qualita2ve
research?

6. *Catherine
Boden
Brooke
Billman
Lorely
Ambriz
Andrea
Miller-­‐Nesbitt
Martin
Morris
Andrew
Booth
Abby
Adamczyk
Anne
Woznica
Keith
Engwall
Rienne
Johnson
Betsy
Clark

7. Search
Databases:
PubMed,
Embase,
ACM,
LISA,
LISTA,
ERIC,
Web
of
Science,
WorldCat
Date
limits:
Ar2cles
• 2000
to
May
2014
Books
• 2005
to
May
2014

8. Preliminary results
Records
aTer
duplicates
removed
(n=3910)
Records
screened
(n=3910)
Full-­‐text
ar2cles
assessed
for
eligibility
(n=165)
Ar2cles
included
in
qualita2ve
analysis
(n=26*)
Data
visualiza2on
(n=7)
Data
mining
(n=9)
Data
management
(n=24)
Records
excluded
in
2tle
abstract
screen
(n=3745)
Full-­‐text
ar2cles
excluded
(n=70*)

9. Data extraction
“Best
fit”
framework
synthesis
Large
result
set
Time
constraints
Large
research
team
MulEple
facets
within
research
quesEon

10. Framework synthesis
• Deduc2ve
process
used
for
systema2c
reviews
• Highly
structured
approach
to
analyzing
qualita2ve
data
• A
priori
framework
is
iden2fied
or
developed
from
a
range
of
sources
• Clearly
defined
themes
in
order
to
ensure
transparency,
consistency
and
speed
of
data
coding

11. ‘Best fit’ framework
synthesis
“The
‘best
fit’
framework
synthesis
method
offered
a
means
to
test,
reinforce
and
build
on
an
exis2ng
published
model,
conceived
for
a
poten2ally
different
but
relevant
popula2on…this
approach
produces
a
rela2vely
rapid,
transparent
and
pragma2c
process.”
(Carroll
et
al.,
2013,
p1)

12. Research
ques2on
Iden2fy
‘best
fit’
frameworks,
conceptual
models
or
theories
Iden2fy
relevant
studies
for
analysis
Generate
a
priori
framework
using
thema2c
analysis
Extract
data
from
included
studies
Code
evidence
from
included
studies
against
a
priori
framework
Create
new
themes
by
doing
thema2c
analysis
on
evidence
that
cannot
be
coded
against
the
framework
Incorporate
new
themes
into
a
priori
framework
to
produce
new
conceptual
model
(Carroll
et
al.,
2013,
p.3)

13. ‘Best fit’ framework(s)
synthesis
• DigCCurr
Data
management
• TBD
Data
mining
Data
visualiza2on
• TBD

14. Data extraction form

15. Data extraction form
“what
are
the
competencies
described
in
the
ar2cle
for
designing
tools
that
support
archival
storage?”
“what
are
the
competencies
in
the
ar2cle
for
designing
tools
for
valida2on
and
quality
control
of
digital
objects/packages?”
…
“descrip2on
of
competencies
relevant
to
the
design
of
tools
for
data
management
that
did
not
fit
in
the
categories
above”
“there
is
insufficient
data
to
be
extracted
with
regard
to
research
data
management”
“record
any
obvious
issues
about
study
quality”

16. Challenges
• Iden2fying
appropriate
frameworks
• Lack
of
granularity
• Missing
various
concepts
(especially
tools)
• Did
not
adequately
address
‘competencies’
• Maintaining
objec2vity
• Ensuring
we
do
not
force
data
into
the
a
priori
framework

17. Next steps
• New
themes
iden2fied
• Relevance
of
data
management,
mining
or
visualiza2on
frameworks
to
librarians’
roles
• Applica2on
of
‘best
fit’
framework
methodology
to
LIS
research

18. Projected outcome
Generate
an
evidence-­‐based
model,
that
iden2fies
the
competencies
required
of
librarians
involved
in
the
design
of
tools
used
for
data
management,
mining
or
visualiza2on.

19. Selected references
Barnek-­‐Page,
E.,
&
Thomas,
J.
(2009).
Methods
for
the
synthesis
of
qualita2ve
research:
a
cri2cal
review.
BMC
Medical
Research
Methodology,
9(1),
1-­‐11.
Carroll,
C.,
Booth,
A.,
&
Cooper,
K.
(2011).
A
worked
example
of
"best
fit"
framework
synthesis:
a
systema2c
review
of
views
concerning
the
taking
of
some
poten2al
chemopreven2ve
agents.
BMC
medical
research
methodology,
11(29).
Carroll,
C.,
Booth,
A.,
Leaviss,
J.,
&
Rick,
J.
(2013).
"Best
fit"
framework
synthesis:
refining
the
method.
BMC
medical
research
methodology,
13(1),
37.
Dixon-­‐Woods,
M.
(2011).
Using
framework-­‐based
synthesis
for
conduc2ng
reviews
of
qualita2ve
studies.
BMC
medicine,
9(1),
39.
Eldredge,
J.
D.,
Ascher,
M.
T.,
Holmes,
H.
N.,
&
Harris,
M.
R.
(2012).
The
new
Medical
Library
Associa2on
research
agenda:
final
results
from
a
three-­‐phase
Delphi
study.
J
Med
Libr
Assoc,
100(3),
214-­‐218.
doi:
10.3163/1536-­‐5050.100.3.012
Eldredge,
J.
D.,
Harris,
M.
R.,
&
Ascher,
M.
T.
(2009).
Defining
the
Medical
Library
Associa2on
research
agenda:
methodology
and
final
results
from
a
consensus
process.
J
Med
Libr
Assoc,
97(3),
178-­‐185.
Ritchie,
J.,
&
Spencer,
L.
(1994).
Qualita2ve
data
analysis
for
applied
policy
research.
In
A.
Bryman
&
R.
G.
Burgess
(Eds.),
Analyzing
qualitaNve
data
(pp.
173-­‐194).
London;
New
York:
Routledge.

20. Questions?
Andrea
Miller-­‐Nesbi
McGill
University,
Montreal,
QC
andrea.miller-­‐nesbik@mcgill.ca