Josh Schnell's presentation from the ORCID and Casrai joint conference, Barcelona, May 2015

1. Real and Perceived Barriers to
Research Assessment
Parallel Session 1.2
ORCID-CASRAI JOINT CONFERENCE
BARCELONA, SPAIN
18 MAY 2015
FROM LOGIC MODEL TO
DATA MODEL

2. 22015 ORCID – CASRAI Joint Conference Barcelona, Spain

3. 32015 ORCID – CASRAI Joint Conference Barcelona, Spain

4. Barriers to Research Assessment
1. Program managers are not familiar with evaluation concepts or do
not have the capacity to carry out evaluation
2. Evaluation is not top-of-mind at the program design stage
3. Insufficient operating budget is available to carry out evaluation
4. Data to support evaluation are not ready-made
5. Burden on the R&D community to support evaluation
42015 ORCID – CASRAI Joint Conference Barcelona, Spain

5. NIH/NCI/CSSI/
OPSO
Established in
2009
$150M, $10M
U54 Research
Grants
Physical Sciences – Oncology Centers
Program (PS-OC)
• To unite the fields of physical
science with cancer biology
and oncology
• To develop trans-disciplinary
teams and infrastructure
• To generate new knowledge
and catalyze new fields of
study
Program Goals
• Twelve centers were funded 2009-
2014, U54 research grants
• 150 main investigators from the
fields of physics, mathematics,
chemistry, engineering, cancer
biology and clinical oncology
• 110 Institutions involved across the
US
PS-OC Network
2015 ORCID – CASRAI Joint Conference Barcelona, Spain 5

6. 6Confidential

7. 2015 ORCID – CASRAI Joint Conference Barcelona, Spain 7

8. Evolution of data available for to support PSO
program management
Manual data
extraction,
organization,
deduplication and
visualization in Excel
Difficult to track
individuals’
contributions
over time
No ability to
collectively
search progress
reports
Data are
entered
through an
intuitive web-
based
interface
Identify data
relationships
Data
deduplication
Flexible,
unified search
On-demand
Bar, pie, and
line charts and
network
graphs
One-click
tabular export
for data behind
graphs or
“report card”
tables
Data can be visualized
by network, center,
person or time, for a
total of over 100
possible charts and
graphs
At-a-glance view of
research output
• Personnel +2,900
• Publications +2,300
• Collaborations +1,900
• Conferences +4,200
• Workshops +400
Manual Data
Analysis Before
iTRAQR
Automated
Data Analysis
With iTRAQR
2015 ORCID – CASRAI Joint Conference Barcelona, Spain 8

9. Deduplication
1. Collaboration
2. Course
3. Funding
4. Meeting
5. Patent
6. Publication
7. Training
Transition
8. Workshop
9. Exchange
10. Project
11. Person
The value of structured data:
clarity, communication and change
2015 ORCID – CASRAI Joint Conference Barcelona, Spain 9

10. Lessons learned for overcoming barriers
1. Start with your evaluation logic model and translate it to your data
model
2. Think about the level of analysis
a. Analyzing subprojects activities at outputs would have been impossible
without iTRAQR
b. Understanding people involved beyond key personnel
3. Data structure is key
4. Have a flexible approach to evaluation (adjust based on findings
using initial data)
102015 ORCID – CASRAI Joint Conference Barcelona, Spain

11. Individual-Level
• Publications
• Patents
• Grants (NIH, other)
• Science Awards (innovative,
translational, training)
• Clinical Trials
• Conference presentations
• Courses and workshops taught
• Trainee disciplines
Center-Level
• Cost, content and people
involved in research projects, pilot
projects and cores
• Stage, content and people
involved in collaborations
• Datasets, techniques,
technologies and bio-specimens
generated and utilized
• Enumeration and content of
transdisciplinary team science
activities
Network-Level
• Cost, content and people
involved in trans-network projects
and outside network pilot projects
• Stage, content and people
involved in collaborations
• Datasets, techniques,
technologies and bio-specimens
generated and utilized
• People and centers involved in
trainee exchanges
• Location and content of
outreach activities
Inputs and Activities Outputs Outcomes
Relative to Comparison Groups
Generated Robust Collaborations that Resulted in Significant Transdisciplinary
Research
• Accelerated the formation of a greater quantity of transdisciplinary
collaborations
• Accelerated the creation of a greater quantity of field convergent research
• Communicated effectively across disciplines to form optimal team sizes
• Effectively contributed to team based activities and outreach
Connected Physical Sciences Perspectives with Clinical Research
• Accelerated the formation of a greater quantity of collaborations among
physical and physician scientists
• Reduced the time between the appearance of a physical sciences perspective
or technology to its application in translational research
• Acted as key investigators leading a convergence of physical sciences
perspectives within translational research and motivating transdisciplinary
translational research
Bridged Oncology Research Gaps
• Accelerated the generation of innovative and impactful transdisciplinary
solutions to outstanding questions in oncology (e.g. integrated
transdisciplinary datasets, technologies and bio-specimens, prominently
positioned in citation networks and commercialized cancer-relevant patented
technology)
Trained a New Generation of Transdisciplinary Scientists
• Conducted a greater quantity of transdisciplinary training activities
• Attracted a greater volume of training grant applications to the PS-OC program
• Graduated a greater quantity of transdisciplinary scientists
• Accelerated the trainee development path toward a career in physical
sciences-oncology
Generated a Sustainable Transdisciplinary Infrastructure
• PS-OC alumni sustained a transdisciplinary perspective by integrating team
science into their infrastructure and attracting new investigators to the field
• Motivated the formation of other inter-/intra- national programs promoting
physical sciences perspectives in cancer research
PS-OC Program Logic Model: Dec 2013
Network-Level
• Coordinate Expertise
 Trans-network Projects
 Physical or virtual
infrastructure
 Integrative training
 Data Coordinating Center
 Research Contracts to further
support clinical translation,
cross-validation and
integration of datasets,
techniques, technologies, bio-
specimens
• Communicate with PS-OC and
Broader Research Community
Center-Level
• Primary leading physical
scientist and cancer researcher
• Research framework: 3-5
projects
• Shared Resources: 1-3 non-
redundant core facilities
• Pilot Projects
• Transdisciplinary lectures,
workshops, working groups,
courses
Individual-Level
• Research findings: pre-award
publications, grants, patents,
clinical trials and business
development
• Research discipline
• Organization associations
(location, Title/Rank,
department)
• Degrees received
• Other demographics

12. Evaluation is only as good as the data
available
• Program design and management is enhanced by early evaluation
design and evaluation efforts
• Data are not infallible and should be part of a holistic evaluation
approach as well as close engagement with program participants
• Data do not exist today to measure all of your program goals
• Careful consideration for what actions can be taken following the
evaluation should help to prioritize data collection
2015 ORCID – CASRAI Joint Conference Barcelona, Spain 12

13. Acknowledgments
Nicole Moore, ScD
Program Director
NCI Division of Cancer Biology
Physical Sciences-Oncology
Unni Jensen, PhD
Sr Scientific Analyst
Thomson Reuters
Jodi Basner, PhD
Scientific Analyst
Thomson Reuters
132015 ORCID – CASRAI Joint Conference Barcelona, Spain

14. THANK YOU

15. Systems to link research to outputs and
outcomes
152015 ORCID – CASRAI Joint Conference Barcelona, Spain