Presented at ASIS&T 2009 in the student awards section. The presentation contains an overview of my dissertation proposal, as 2009 winner of the Thomson Reuters Information Science Doctoral Dissertation Proposal Scholarship, administered by the ASIS&T Information Science Education Committee

1. Foundational studies for measuring the
impact, prevalence, and patterns
of publicly sharing
biomedical research data
Heather Piwowar
Department of Biomedical Informatics
University of Pittsburgh

2. Sharing research data
http://upload.wikimedia.org/wikipedia/commons/7/76/PeptideMSMS.jpg; http://en.wikipedia.org/wiki/Image:Helices.png; http://en.wikipedia.org/wiki/
Image:Heatmap.png; http://en.wikipedia.org/wiki/Image:Microarray2.gif;
http://zellig.cpmc.columbia.edu/medlee/demo/; htp://www.plosone.org/article/fetchArticle.action?articleURI=info:doi/10.1371/journal.pone.0000441

3. Sharing research data
http://upload.wikimedia.org/wikipedia/commons/7/76/PeptideMSMS.jpg; http://en.wikipedia.org/wiki/Image:Helices.png; http://en.wikipedia.org/wiki/
Image:Heatmap.png; http://en.wikipedia.org/wiki/Image:Microarray2.gif;
http://zellig.cpmc.columbia.edu/medlee/demo/; htp://www.plosone.org/article/fetchArticle.action?articleURI=info:doi/10.1371/journal.pone.0000441

4. Sharing research data
PAST MEDICAL HISTORY:
Past medical history showed she had
superficial phlebitis times two in the past, had
non-insulin dependent diabetes mellitus for
four years.
She had been hypothyroid for three years.
HISTORY OF PRESENT ILLNESS:
The patient is a 58-year-old female, …
http://upload.wikimedia.org/wikipedia/commons/7/76/PeptideMSMS.jpg; http://en.wikipedia.org/wiki/Image:Helices.png; http://en.wikipedia.org/wiki/
Image:Heatmap.png; http://en.wikipedia.org/wiki/Image:Microarray2.gif;
http://zellig.cpmc.columbia.edu/medlee/demo/; htp://www.plosone.org/article/fetchArticle.action?articleURI=info:doi/10.1371/journal.pone.0000441

5. Sharing research data
PAST MEDICAL HISTORY:
Past medical history showed she had
superficial phlebitis times two in the past, had
non-insulin dependent diabetes mellitus for
four years.
She had been hypothyroid for three years.
HISTORY OF PRESENT ILLNESS:
The patient is a 58-year-old female, …
http://upload.wikimedia.org/wikipedia/commons/7/76/PeptideMSMS.jpg; http://en.wikipedia.org/wiki/Image:Helices.png; http://en.wikipedia.org/wiki/
Image:Heatmap.png; http://en.wikipedia.org/wiki/Image:Microarray2.gif;
http://zellig.cpmc.columbia.edu/medlee/demo/; htp://www.plosone.org/article/fetchArticle.action?articleURI=info:doi/10.1371/journal.pone.0000441

6. Sharing research data
PAST MEDICAL HISTORY:
Past medical history showed she had
superficial phlebitis times two in the past, had
non-insulin dependent diabetes mellitus for
four years.
She had been hypothyroid for three years.
HISTORY OF PRESENT ILLNESS:
The patient is a 58-year-old female, …
http://upload.wikimedia.org/wikipedia/commons/7/76/PeptideMSMS.jpg; http://en.wikipedia.org/wiki/Image:Helices.png; http://en.wikipedia.org/wiki/
Image:Heatmap.png; http://en.wikipedia.org/wiki/Image:Microarray2.gif;
http://zellig.cpmc.columbia.edu/medlee/demo/; htp://www.plosone.org/article/fetchArticle.action?articleURI=info:doi/10.1371/journal.pone.0000441

7. Sharing research data
PAST MEDICAL HISTORY:
Past medical history showed she had
superficial phlebitis times two in the past, had
non-insulin dependent diabetes mellitus for
four years.
She had been hypothyroid for three years.
HISTORY OF PRESENT ILLNESS:
The patient is a 58-year-old female, …
http://upload.wikimedia.org/wikipedia/commons/7/76/PeptideMSMS.jpg; http://en.wikipedia.org/wiki/Image:Helices.png; http://en.wikipedia.org/wiki/
Image:Heatmap.png; http://en.wikipedia.org/wiki/Image:Microarray2.gif;
http://zellig.cpmc.columbia.edu/medlee/demo/; htp://www.plosone.org/article/fetchArticle.action?articleURI=info:doi/10.1371/journal.pone.0000441

8. http://www.flickr.com/photos/75166820@N00/5318468/

9. Shared data benefits science
Verify
Understand
Extend
Explore
Combine
Synergize
Train
Reduce

10. But... costly for authors
Find
Organize
Document
Deidentify
Format
Decide
Ask
Submit
Answer questions
Worry about mistakes being found
Worry about data being misinterpreted
Worry about being scooped
Forgo money and IP and prestige???

11. As a result, policy makers have spent
lots of time and money ....
http://www.flickr.com/photos/johnnyvulkan/381941233/
http://www.flickr.com/photos/tonivc/2283676770/

12. ... on initiatives, requests,
requirements, and tools
Funder data sharing requirements
Journal requirements and requests
Databases
Data sharing collaboration grids
Standards
Editorials, letters to the editor, discussion....

13. http://www.flickr.com/photos/mesh/14102209/

14. lots of data sharing!
http://www.genome.jp/en/db_growth.html

15. but how much isn’t
shared?
what isn’t shared?
who isn’t sharing it?
why not?
how much does it matter?
what can we do
about it?

16. you can not manage
what you do not measure
http://www.flickr.com/photos/archeon/2941655917/

17. http://www.flickr.com/photos/archeon/2941655917/

18. Related research
Data usually collected via surveys
and/or manual audits
http://www.flickr.com/photos/jima/606588905/

19. Models of data and knowledge
sharing

20. Andriessen. Conditions for the willingness to share knowledge, 2006.

21. Harder. SMG WP 6/2008 .

23. Cabrera and Cabrera. Int J of HR Mgmt. 2005.

25. Kuo. JASIST. 2008.

26. Limitations of the related research
• manual audits: small sample sizes
• surveys: few variables + self-reporting bias
• not much focus on measuring demonstrated behavior
• not much focus on rewards
• not much focus on policy
• not much focus on biomedical data other than
DNA sequences

27. Needed:
a study of data sharing behaviour and impact
that includes
• a measurement of demonstrated behavior
• policy variables
• estimate of rewards
• a broad and deep selection of data creation instances

28. Aim 1: Does sharing have benefit for
those who share?
Aim 2: Can sharing and withholding be
systematically measured?
Aim 3: How often is data shared?
What predicts sharing?
How can we model sharing behavior?

29. Scope of proposed study
studies
Published studies with English full text available in
a centralized portal
variables for examination
extracted from Medline and other sources

30. Microarray data
http://en.wikipedia.org/wiki/DNA_microarray
http://en.wikipedia.org/wiki/Image:Heatmap.png
http://commons.wikimedia.org/wiki/
File:DNA_double_helix_vertikal.PNG

31. http://farm3.static.flickr.com/2146/2389590651_9bbcc9d07e.jpg

32. Aim 1

33. Aim 1: Does sharing have benefit
for those who share?
http://www.flickr.com/photos/sunrise/35819369/

34. Aim 1: Does sharing have benefit
for those who share?
Benefit of value: Citations.

35. Aim 1: Does sharing have benefit
for those who share?
dataset
85 cancer microarray trials published in 1999-2003,
as identified by Ntzani and Ioannidis (2003)
citations
ISI Web of Science Citation index, citations from
2004-2005
data sharing locations
Publisher and lab websites, microarray databases,
WayBack Internet Archive, Oncomine
statistics
Multivariate linear regression

36. Aim 1: Does sharing have benefit
for those who share?

37. Aim 1: Does sharing have benefit
for those who share?
Note the
logarithmic
scale

38. Aim 1: Does sharing have benefit
for those who share?
In multivariate regression, we found studies that had
made their data publicly available received 69% more
citations than similar studies that did not share their
data (95% confidence interval: 18% to 143%)
Piwowar, Day and Fridsma (2007) Sharing Detailed
Research Data Is Associated with Increased Citation
Rate. PLoS ONE 2(3): e308

39. Aim 1 conclusion: data sharing has a
benefit for sharers

40. Next: What factors predict sharing?
http://www.flickr.com/photos/ryanr/142455033/

41. Next: What factors predict sharing?
Can I use the same methods of Aim 1
to choose studies and determine data sharing status?
http://www.flickr.com/photos/ryanr/142455033/

42. Next: What factors predict sharing?
Can I use the same methods of Aim 1
to choose studies and determine data sharing status?
No, those methods donʼt scale to identify or classify
enough datapoints.
http://www.flickr.com/photos/ryanr/142455033/

43. Aim 2

44. Need automated methods to:
Identify studies that generate datasets that
could potentially be shared (Aim 2a)
Determine which of these have in fact been
shared (Aim 2b)

45. Aim 2a: Identify studies that create
gene expression microarray data
http://www.flickr.com/photos/lofaesofa/248546821/

46. Aim 2a: Identify studies that create
gene expression microarray data
Easy, via MeSH indexing terms?
gene expression profiling and/or
microarray analysis
Unfortunately, has neither high recall nor precision.

47. Aim 2a: Identify studies that create
gene expression microarray data
Instead, look for wetlab methods in full text:
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1522022&tool=pmcentrez
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1590031&tool=pmcentrez
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1482311&tool=pmcentrez#id331936
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2082469&tool=pmcentrez
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=126870&tool=pmcentrez#id442745

48. Aim 2a: Identify studies that create
gene expression microarray data
And query the full text through full-text query portals:

49. Aim 2a: Identify studies that create
gene expression microarray data
query development
Use supervised natural language processing
techniques on a corpus of Open Access articles
query evaluation
400 studies that created gene expression microarray
data, as identified by Ochsner et al (2008)
goal
>90% precision, and sufficient recall to retrieve >1250
articles

50. Aim 2b

51. Aim 2b: Identify studies that share
their expression microarray data
http://www.flickr.com/photos/dcassaa/422261773/

52. Aim 2b: Identify studies that share
their expression microarray data

53. Aim 2b: Identify studies that share
their expression microarray data

54. Aim 2b: Identify studies that share
their expression microarray data
pmc_gds[filter]
+ text processing on
ArrayExpress website
Enough? Unbiased?

55. Aim 2b: Identify studies that share
their expression microarray data
reference standard
200 the 400 studies that created gene expression
microarray data have shared their microarray data, as
identified by Ochsner et al (2008)
goal
Establish that filter has >70% recall with an unbiased
representation of MeSH terms, dataset size, and
dataset species

56. Aim 3

57. Aim 3 – How often is data shared?
What predicts sharing?
How can we model sharing behavior?
http://www.flickr.com/photos/ryanr/142455033/

58. Aim 3a: Prevalence of data sharing

59. Aim 3a: Prevalence of data sharing
PubMed Created
Portal
ID data?
234 PMC Yes
345 HighPr Yes
456 Scirus Yes
567 PMC Yes
678 PMC Yes
789 HighPr No
890 PMC No
901 ‐ ?

60. Aim 3a: Prevalence of data sharing
PubMed Created
Portal
ID data?
234 PMC Yes
345 HighPr Yes
456 Scirus Yes
567 PMC Yes
678 PMC Yes
789 HighPr No
890 PMC No
901 ‐ ?

61. Aim 3a: Prevalence of data sharing
PubMed Created
Portal
ID data?
234 PMC Yes
345 HighPr Yes
456 Scirus Yes
567 PMC Yes
678 PMC Yes

62. Aim 3a: Prevalence of data sharing
PubMed Created Shared
Portal
ID data? data?
234 PMC Yes Yes
345 HighPr Yes Yes
456 Scirus Yes Yes
567 PMC Yes NO
678 PMC Yes NO

63. Aim 3a: Prevalence of data sharing
PubMed Created Shared
Portal
ID data? data?
234 PMC Yes Yes
345 HighPr Yes Yes
456 Scirus Yes Yes
567 PMC Yes NO
678 PMC Yes NO
Prevalence = Number with Shared data
Number with Created data

64. Aim 3b: Correlates with data sharing

65. Aim 3b: Correlates with data sharing
Covariates
PubMed Created Shared
Portal
ID data? data?
234 PMC Yes Yes
345 HighPr Yes Yes
456 Scirus Yes Yes
567 PMC Yes NO
678 PMC Yes NO

66. Aim 3b: Correlates with data sharing
Features to include:
• Does the journal have a data sharing policy?
• Is the study funded by the NIH?
• Is it subject tot the NIH data sharing plan
requirement?
• Number of authors
• Journal impact factor
• Are the experimental samples from humans?
• Disease of study
• Year of publication
• …

67. Aim 3b: Correlates with data sharing
Covariates
PubMed Created Shared Journal NIH #
Portal ...
ID data? data? policy funds? authors
234 PMC Yes Yes strong yes 2
345 HighPr Yes Yes weak yes 5
456 Scirus Yes Yes weak no 6
567 PMC Yes NO strong yes 5
678 PMC Yes NO strong no 2

68. Aim 3b: Correlates with data sharing
Univariate odds ratios
Multivariate logistic regression

69. Aim 3b: Correlates with data sharing
Covariates
PubMed Created Shared Journal NIH #
Portal ...
ID data? data? policy funds? authors
234 PMC Yes Yes strong yes 2
345 HighPr Yes Yes weak yes 5
456 Scirus Yes Yes weak no 6
567 PMC Yes NO strong yes 5
678 PMC Yes NO strong no 2
Journal policy? NIH funded? # authors ...
Shared data?

70. Aim 3c: Model of data sharing

71. Aim 3c: Model of data sharing
Covariates
PubMed Created Shared Journal NIH #
Portal ...
ID data? data? policy funds? authors
234 PMC Yes Yes strong yes 2
345 HighPr Yes Yes weak yes 5
456 Scirus Yes Yes weak no 6
567 PMC Yes NO strong yes 5
678 PMC Yes NO strong no 2

72. Aim 3c: Model of data sharing
Exploratory factor analysis

73. Aim 3c: Model of data sharing
Covariates
PubMed Created Shared Journal NIH #
Portal ...
ID data? data? policy funds? authors
234 PMC Yes Yes strong yes 2
345 HighPr Yes Yes weak yes 5
456 Scirus Yes Yes weak no 6
567 PMC Yes NO strong yes 5
678 PMC Yes NO strong no 2
Mandates Amount of
Collaboration
Shared data? ...

74. Aim 3c: Model of data sharing
Covariates
PubMed Created Shared Journal NIH #
Portal ...
ID data? data? policy funds? authors
234 PMC Yes Yes strong yes 2
345 HighPr Yes Yes weak yes 5
456 Scirus Yes Yes weak no 6
567 PMC Yes NO strong yes 5
678 PMC Yes NO strong no 2
Mandates Amount of
Weak Collaboration
Strong Shared data? ...

75. http://www.flickr.com/photos/donjuanna/322798429/

76. Limitations
• Association does not imply causation
• Important influences will be missed due to focus on
measurable variables
• Some derived variables involve many estimates and
assumptions
• Only considering public sharing in primary
centralized databases
• Only one datatype
• Only research studies made available in full-text
portals

77. Risks and contingency plans
NLP performance may be inadequate
supplement with manual annotating via Mechanical Turk
Author ambiguity may introduce extreme outliers
use Author-ity (Smalheiser and Torvik, 2005) for name
disambiguation
Unable to derive a robust exploratory factor model
try other clustering techniques
Several variables may be unexpectedly difficult to
extract and cross-references
if not essential, defer analysis of that variable

78. Current status
Aim 1: Does sharing have benefit for
those who share?
Aim 2: Can sharing and withholding be
systematically measured?
ete d.
Aim 3: How often is data shared? ction
om pl
lot c
What predicts sharing? set c olle
pi ata behavior?
ll d
How can we model sharing
fu
No w:

79. Anticipated contributions
• Published assessment of the observed and
measured rewards, prevalence, and patterns of
gene expression microarray dataset sharing
• Publicly available dataset associating microarray
study publications with data sharing status
• Generalizable approach for developing practical,
real-world information retrieval using
centralized full-text query portals
• Preliminary model of data sharing behaviour
based on this large dataset

80. Future work
• Identify and model data reuse
• Citation analysis of the large cohort
• Supplement with survey responses
http://www.flickr.com/photos/cogdog/123072/

81. Data sharing plan
I post my data, code, and statistical scripts at
http://www.dbmi.pitt.edu/piwowar
Share yours too!
http://www.flickr.com/photos/myklroventine/892446624/

82. Thanks to:
➡ the NLM for funding training grant 5 T15 LM007059-22
➡ the Dept of Biomedical Informatics at the U of Pittsburgh
➡ my committee
Dr Wendy Chapman Biomed Informatics
Dr Ellen Detlefsen iSchool
Dr Madhavi Ganapathiraju Bioinformatics
Dr Brian Butler Katz School of Business
Dr Gunther Eysenbach U of Toronto, Health Policy
Mgmt and Evaluation

84. Funder Journal Investigator Institution Study
Is research data shared
after publication?
aim

85. Prevalence of data withholding
via surveys
self-reported denying a request in last 3 years
trainees self-reported denying a request
been denied access to data, materials, code
authors “not able to retrieve raw data”
not willing to release data
0% 10% 20% 30% 40%
Campbell et al. JAMA. 2002.
Kyzas et al. J Natl Cancer Inst. 2005.
Vogeli et al. Acad Med. 2006.
Reidpath et al. Bioethics 2001.

86. Self‐reported reasons for data
withholding
sharing is too much effort
want student or jr faculty to publish more
they themselves want to publish more
cost
industrial sponsor
confidentiality
commercial value of results
0% 20% 40% 60% 80%
Campbell et al. JAMA 2002.

87. Correlates with self‐reported data
withholding
industry involvement
perceived competitiveness of field
male
sharing discouraged in training
human participants
academic productivity
0 1 2 3
Blumenthal et al. Acad Med. 2006