Presentation made to the IOM Forum on Drug Discovery, Development, and Translation to explore the possibility of metrics that evaluate the performance of biomedical research consortia

1. Strengthening the
Effectiveness and Impact of
Collaboration Through Metrics
Mark D. Lim, PhD
Forum on Drug Discovery, Development, and Translation
Institute of Medicine
October 7, 2014

2. research-by-consortium
 Shared scientific challenge
 Widely-usable tool
 Virtual team to create / qualify

3. consortiapedia.fastercures.org
Operational Framework Landscape
Sci. Trans. Medicine, June 2014
http://bit.ly/STMConsortia
Database
Planned release:
end of 2014

4. Metrics for consortia?
We need to value collective successes of a consortium…
and get rewarded for that.
Eric Green, NHGRI Director
Whatever can be done in qualifying how consortium work, how
quality is being measured and recognized, would help big
companies like ours to work in these consortium.
Marc Bonnefoi, Sanofi
FasterCures Hill Briefing, July 2014
Considerations:
 Objective measurements – performance and outcome
 Apples and apples - normalized across consortia
 Suite of metrics by project type?
 Application (biomarker development, clinical trial methods, etc)
 Therapeutic area
 Stage of innovation
 Others?

5. Since we last met….
Conversations
• Consortium leaders and participants
• Pharmaceutical companies
• FDA
• Collaboration management experts
Presentations
• Science of Team Science
• Pharmaceutical campus
Other
• Hill Briefing on consortia
qualitative
metrics…asking
participants what the
vision/mission is…
An evolving conversation…
what are indicators
for future success?
is this consortium
doing the best it can?
what are common
challenges that all
consortia are facing?

6. Where we want to go…
 Reality check on value of framework
 General consensus on:
 Design
 Implementation
 Optimization
 Who are implementation partners?
 What is the output? What does success look like?
 Evidence to inform best practices?

7. Metrics of collaboration?
Retrospective Prospective
• Return on investment
• Impact on organization
• Impact on society
• Improved efficiency
• Focus vs flexibility
Output and outcomes
Sponsors/Participants
• What did we get from the collaboration?
• Would we participate in a similar collaboration again?
Managers/Organizers
• Were there lessons learned?
• How do we design/execute a better collaboration?

8. Value of outcomes: who is asking?
Government • public health
• regulatory science
• de-risk innovation
• economic growth
• state-of-science research guidance documents
Industry • accelerate pipeline
• new therapeutic area
• access resources
• de-risk innovation
• access intellectual
capital
Academia • access resources
• opportunities for
publications
• training opportunities
• identify collaborators
Patient
organizations
• accelerate pipelines
• advance basic
research
• de-risk medical
product development
Consortium
researchers
• simplify day jobs
• access resources
• networking
• training / education

9. Output to outcome metrics
Non-competitive biomarkers
Clinical trial designs and processes
Data standards
Best practices / procedures
Tools (PRO, prediction, data-sharing)
Output
Implementation / adoption
IP – filed / awarded / licensed
Policy change / regulatory science
Publications
Job / company creation
Impact
De-risked innovation
Decreased R&D cost/time
Meaningful data-sharing
Improved clinical trials
Improved human health
Outcome
* Adapted from Critical Path Institute and Innovative Medicines Initiative

10. Metrics of collaboration?
Retrospective Prospective
• Return on investment
• Impact on organization
• Impact on society
• Improved efficiency
• Focus vs flexibility
Processes
Sponsors/Participants
• Is the collaboration on-track and focused?
• Are resources available/leveraged?
Managers/Organizers
• How can we do better?
• Are we continuing to address an unmet need?

11. Processes & Team Performance
Execute
Wooten, Univ. Houston
Tuckman & Jensen, Group and Organizational Studies (1977)
Burke et al., J. Applied Psychology (2006)
Hall et al, Trans Behavioral Med (2012)
Identifying shared challenge
Consensus solution
Framework for collaboration
Defining success
Convene
Defining accomplishments
Integrating deliverables
Communicating
Tracking adoption/impact
Transition
Participant/sponsor buy-in
Maintaining vision/focus
Work stream monitoring
Conflict resolution

12. Discussion points
?
 Gut reaction?
 Structuring discussion to develop framework?
• How? (retrospective/prospective)
• What? (end-user)
• Who? (stakeholders)
• Where? (host)
 Next-steps
• Follow-up with consortia, industry, FDA
• Partnering For Cures – Nov 16-18 (NYC)

13. Seeking insights & expertise
consortiapedia.fastercures.org
mlim@fastercures.org