Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this document? Why not share!

Like this? Share it with your network

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Health Research Policy and Systems BioMed Central Review Open Access Developing a conceptual framework for an evaluation system for the NIAID HIV/AIDS clinical trials networks Jonathan M Kagan*1, Mary Kane2, Kathleen M Quinlan2, Scott Rosas2 and William MK Trochim3 Address: 1Division of Clinical Research, NIAID/NIH, 6700 B Rockledge Drive, Room 1102, MSC 7609, Bethesda, Maryland 20892-7609, USA, 2Concept Systems, Inc, 136 East State St, Ithaca, New York 14850, USA and 3Department of Policy Analysis and Management, Cornell Office for Research on Evaluation, Martha Van Rensselaer Hall, Cornell University, Ithaca, New York 14853-4401, USA Email: Jonathan M Kagan* - jkagan@niaid.nih.gov; Mary Kane - mkane@conceptsystems.com; Kathleen M Quinlan - kquinlan@conceptsystems.com; Scott Rosas - srosas@conceptsystems.com; William MK Trochim - wmt1@cornell.edu * Corresponding author Published: 21 May 2009 Received: 12 May 2008 Accepted: 21 May 2009 Health Research Policy and Systems 2009, 7:12 doi:10.1186/1478-4505-7-12 This article is available from: http://www.health-policy-systems.com/content/7/1/12 © 2009 Kagan et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Globally, health research organizations are called upon to re-examine their policies and practices to more efficiently and effectively address current scientific and social needs, as well as increasing public demands for accountability. Through a case study approach, the authors examine an effort undertaken by the National Institute of Allergy & Infectious Diseases (part of the National Institutes of Health, Department of Health & Human Services, United States Government) to develop an evaluation system for its recently restructured HIV/AIDS clinical trials program. The challenges in designing, operationalizing, and managing global clinical trials programs are considered in the context of large scale scientific research initiatives. Through a process of extensive stakeholder input, a framework of success factors was developed that enables both a prospective view of the elements that must be addressed in an evaluation of this research and a current state assessment of the extent to which the goals of the restructuring are understood by stakeholders across the DAIDS clinical research networks. Introduction This manuscript describes the process and progress to date Around the globe, health research organizations are in developing a systematic approach to support the evalua- examining ways of designing, managing and evaluating tion of a global human immunodeficiency virus (HIV) clin- their programs so they address local and international ical trials research program. Further, it analyzes and draws health needs and achieve maximum value for their inferences from this specific work that may have broad investments. These practical concerns are sparking inter- applicability for the management, operationalization, and est in research on health systems research, in order to evaluation of other large-scale research endeavors. improve results and create an evidence base that can guide policy-makers on the management of health The specific case study highlighted here is situated in the research [1]. recent restructure of the world's largest human immuno- Page 1 of 16 (page number not for citation purposes)
  • 2. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 deficiency virus/acquired immunodeficiency syndrome ciencies and synergism inherent in these systems of (HIV/AIDS) clinical trials program. We begin by reviewing research are expected to positively impact population the program's context, its scientific goals and objectives, health and behavior by producing innovative, relevant, rationale for the restructure, and how the factors that and timely research. These broader goals, and the opera- drove the restructure were addressed in the restructuring tional activities that support them, must be taken into plan. We then focus on the development of a conceptual account when developing evaluation approaches [3]. framework as the foundation of a larger evaluation sys- tem, emphasizing the process and results of a stakeholder- Thus, the multiplicity of goals and stakeholder agendas driven effort to describe the success factors for the newly which these research enterprises simultaneously pursue, restructured program. We discuss the resulting framework poses substantial management, implementation and eval- in the context of the program's goals and its implications uation challenges. At the same time, there are growing for the evaluation of other large scale research endeavors. pressures for accountability for federally funded scientific research [8-11], and recent law to include performance Background on the transformation of science monitoring officers. [12,13]. In recent years, there has been a broad transformation of the organization and management of science in the Leading scientific bodies, responding to government United States [2,3]. In addition to traditional support for expectations, have issued their own recommendations for individual scientists working on their own, federally the evaluation of scientific research endeavors [6,14-16]. funded scientific research now emphasizes "big science": This paper explores the management and evaluation chal- large, collaborative research initiatives, with annual budg- lenges associated with the HIV/AIDS clinical research net- ets of $5 million or more. The number of these types of works. Through a case-study approach, we describe a initiatives has increased in recent years [4]. participatory process applied in this setting to construct a conceptual framework, and explore the utility of the One major type of "big science" is clinical research net- framework to shape a system of network evaluation. works of which the HIV/AIDS clinical networks described in this manuscript are a prominent example. There are Background on the NIAID Division of AIDS and nearly 300 clinical research networks in the United States its clinical trials networks and Canada. The majority are funded primarily by the US The Division of Acquired Immunodeficiency Syndrome Government and nearly half carry out clinical trials as their (DAIDS) is one of three extramural scientific divisions of primary activity [5]. Non-trial studies include observational the National Institute for Allergy and Infectious Diseases research, outcomes research or best practice modeling. (NIAID), the second largest of the 27 institutes and cent- ers that comprise the U.S. National Institutes of Health Another type of "big science" is the extramural research (NIH). Established in 1986, the mission of DAIDS is to center structure, first funded in the 1960s. There are now help ensure an end to the HIV/AIDS epidemic by increas- more than 1200 center programs, with each of the ing basic knowledge of the pathogenesis and transmission National Institutes of Health (NIH) institutes using this of the human immunodeficiency virus (HIV), supporting approach [6]. These programs generally take a multi-disci- the development of therapies for HIV infection and its plinary team approach focused on interaction between complications and co-infections, and supporting the basic and clinical researchers to foster translational development of vaccines and other prevention strategies. research. With an annual budget of approximately one billion dol- lars, DAIDS is a major source of funding for biomedical Large research initiatives generally have broader goals research around the world. than traditional individual investigator-initiated grants [7]. In addition to providing scientific outputs, these Beginning with the AIDS Treatment Evaluation Units in multi-million dollar, multi-institutional programs are 1987, DAIDS goal in establishing the HIV/AIDS networks also expected to foster multi-disciplinary teamwork, pro- was to assemble multidisciplinary scientific expertise, and vide more effective support for independently funded create sustainable, reusable capacity and infrastructure to investigators, gain increased attention to a program's support a long-term commitment to clinical trials. The research by the center's home institution, recruit estab- ensuing 19 years witnessed significant scientific advances lished researchers to the program's area of interest, as well as major changes in the global demographics of develop new investigators, expand the education of health HIV/AIDS, presenting both challenges and opportunities professionals and the general public, build scientific infra- for research. In response, the DAIDS networks have structure and demonstrate state-of-the-art prevention, evolved and adapted. An overview of the history and evo- diagnosis and treatment techniques. Ultimately, the effi- lution of the DAIDS networks is shown in Figure 1. Page 2 of 16 (page number not for citation purposes)
  • 3. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 1983 1988 1993 1998 2003 2008 2013 AACTG Adult AIDS Clinical Trials Group ACTG AIDS Clinical Trials Group ATEU AIDS Treatment Evaluation Units AVEG AIDS Vaccine Evaluation Group Terry Beirn Community Programs for CPCRA Clinical Research on AIDS CSG Cooperative Study Groups Evaluation of Subcutaneous Proleukin ESPRIT in a Randomized International Trial HIVNET HIV Network for Prevention Trials HPTN HIV Prevention Trials Network HVTN HIV Vaccine Trials Network International Maternal Pediatric IMPAACT Adolescent AIDS Clinical Trials Group International Network for Strategic INSIGHT Initiatives in Global HIV Trials MTN Microbicide Trials Network PACTG Pediatric AIDS Clinical Trials Group Preparation for AIDS Vaccine PAVE Evaluations Timeline of NIAID’s HIV/AIDS clinical trials networks by U.S. Government fiscal years (October 1 – September 30). Horizontal bars depict networks (or pre network programs). Connections between bars represent time points where a program/network was competitively renewed. Figure 1 Evolution of the NIAID HIV/AIDS clinical trials networks Evolution of the NIAID HIV/AIDS clinical trials networks. Restructuring the HIV/AIDS clinical trials program Further, the following key guiding principles were identi- In 2001, several factors, including the size and complexity fied: of the network enterprise, changes in the HIV/AIDS knowledge base, unabated expansion of the epidemic in Responsiveness the developing world, and fiscal considerations, Maintain a flexible and responsive approach to emerging prompted the DAIDS leadership to reexamine the organi- research challenges that ensures that the highest research zational and funding structure of the networks. Taking priorities are addressed into account these and other factors, the Division began working with investigators, collaborators, advisory com- Efficiency mittees, community groups, and a wide range of stake- Improve efficiency through the shared use of key support holders to help develop the scientific priorities going services (e.g. laboratories, pharmacies), common data ele- forward, and gather input on how best to restructure the ments and harmonized data systems, coordinated speci- clinical trials programs in anticipation of a grant competi- men management, shared/standardized training for tion in 2005. Six scientific priority areas for HIV/AIDS common needs, coordinated clinical product acquisition, clinical trials research emerged from this effort: distribution and provision. • HIV Vaccine Research & Development Coordination Harmonize and/or integrate HIV/AIDS prevention, vac- • Translational Research for Therapeutic Development cine and therapeutic research at the scientific and commu- nity levels, in order to maximize research opportunities • Optimization of Clinical Management and better respond to public health needs • Microbicide Research & Development Capacity Building Build and strengthen HIV/AIDS research capacity, espe- • Mother-to-Child Transmission of HIV cially in resource-limited settings, both domestically and internationally • Prevention Research Page 3 of 16 (page number not for citation purposes)
  • 4. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 Evaluation 4. To help address the likely scenario of emergent high Strengthen evaluation of network science and manage- priority research opportunities that could require ment to help ensure that the networks address the highest timely reordering of scientific priorities, new flexible priority research questions, and function as efficiently and resource allocations were developed. Both RFAs effectively as possible through the identification and described the importance of performance evaluation, implementation of best practices, and linking funding to designation of new 'reserve' funds, and a role for exter- performance. nal advisory bodies to assist in meeting such chal- lenges. To put these scientific and operational priorities into place, DAIDS developed and issued two "Request for In June 2006, NIAID announced funding for the restruc- Application" (RFAs) that differed substantially from pre- tured HIV/AIDS clinical trials networks: vious network solicitations in several ways. • AIDS Clinical Trials Group http://aactg.s-3.com/ 1. Whereas in the past DAIDS issued multiple "dedi- cated" RFAs (one for each clinical trials network it • HIV Prevention Trials Network http://www.hptn.org sought to renew), this time a single Leadership Group RFA was used. Competition through a single RFA • HIV Vaccine Trials Network http://www.hvtn.org highlighted the importance of the linkages between the scientific priorities, the applicability of the guiding • International Maternal Pediatric Adolescent AIDS principles to all applicant networks, and the decision Clinical Trials http://impaact.s-3.com/ not to set a pre-determined funding level for any one (or group) of the six scientific area(s). • International Network for Strategic Initiatives in Global HIV Trials http://insight.ccbr.umn.edu/ 2. Similarly, a single RFA was used to solicit applica- tions for Clinical Trials Units (CTU) seeking funds to • Microbicide Trials Network http://www.mtnstop carry out the future network trials. In previous funding shiv.org cycles, DAIDS had issued RFAs for CTU that were 'spe- cific' to a particular network. The rationale behind the Together, including the network leadership groups and 73 single CTU RFA was to maximize scientific opportu- CTU located around the world, funding for the networks nity and efficiency by allowing investigators (includ- was approximately $285 million in the first year of ing those in resource poor settings) interested in awards. participating in more than one type of clinical trial (e.g. vaccines and treatments) to describe their capa- Evaluation goals for the restructured networks bilities, and how they would coordinate and harmo- DAIDS considers a comprehensive, integrated evaluation nize this research, within a single application. This system a vital element of the new coordinated network approach also supports DAIDS goal to help build HIV/ structure. Prior to restructuring (and currently), evalua- AIDS clinical research capacity by making grant tion activities have been almost exclusively 'network cen- awards directly to institutions in areas where the dis- tric'. Each group's investigator leadership has primary ease has hit the hardest, and where resources are often responsibility for developing and implementing criteria constrained. and processes for assessing the structural components within a single Network, such as the clinical trials units, 3. To stimulate the implementation of a cross-cutting laboratories, data management centers, scientific and and interdisciplinary research agenda, both RFAs resource committees, and protocol teams. In some described four key organizational elements/activities: instances, ad hoc expert review panels have provided 1) a new committee comprised of the Principal Investi- external input and evaluative information. While such gators of the networks; 2) a new central office of HIV/ single network-focused evaluation activities remain very AIDS network coordination; and 3) a new Community important, DAIDS also recognizes the need for a cross-cut- Partners group, to enhance community input on all lev- ting evaluation framework that can support the common els, and assure effective representation of (and timely guiding principles that span the restructured network communication among) the many communities, enterprise (e.g. responsiveness, efficiency, coordination, (domestic and international) within which the net- capacity building). Such a system can assist DAIDS, its works conduct research, and; 4) coordination, commu- investigators and collaborators in ensuring that the high- nication and collaboration across the Institutes and est priority scientific objectives are addressed; while sup- Centers at NIH that co-sponsor or, in other ways, con- porting collaboration, efficiency and research integration tribute the HIV/AIDS clinical research networks. at the cross-network level. While the approach being Page 4 of 16 (page number not for citation purposes)
  • 5. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 developed is designed to build upon and complement DAIDS clinical research networks encompasses several existing within-network evaluation approaches, it will integrated phases. Figure 2 displays the focus and also define criteria and measures that should be applica- sequencing of these phases leading to an evaluation sys- ble across the networks. It will address a variety of factors tem, including preparation, inquiry, framework develop- required for the success of clinical research networks, ment, and the evaluation plan. In this paper we describe including activities and processes based within (as well as the initial phases in this process: the creation of a concep- across) the investigator networks, constituency communi- tual framework and show how it informs and connects to ties, collaborators (e.g. support contractors), and DAIDS the other planning steps. itself (e.g. regulatory, safety, clinical research policy). Engaging stakeholders in constructing an The DAIDS HIV/AIDS clinical trials networks are an exam- evaluation framework ple of a large scale scientific initiative and as such, share There are numerous challenges in developing appropriate much in common with similar ventures, including broad evaluation systems for large research initiatives. These goals, a multidisciplinary focus, and emphasis on team include: ensuring the highest-quality evidence; minimiz- science. However, the HIV/AIDS clinical trials networks ing burden and the intrusiveness of the system; and yield- are unique due to aspects related specifically to the con- ing data that can satisfy a myriad of scientific, managerial, duct of international human clinical research, namely: 1) financial, and regulatory requirements. One of the most the reliance on inter-institutional collaboration among challenging aspects is identifying the goals of a complex multiple clinical research centers and other collaborators research initiative across a wide array of stakeholders, each (e.g., pharmaceutical sponsors, other co-funding agen- with a set of expectations as to what constitutes success. cies); 2) interaction with research constituents/advocates, There are few precedents to guide evaluation planning for including vulnerable populations, and related ethical such efforts. With the expectation that today's scientific issues; 3) compliance with regulatory authorities of mul- research enterprise is collaborative and coordinated at tiple countries which frequently are not harmonized; 4) multiple levels, several sources of input are needed to the need to maintain a consistent and yet humanitarian define the goals of these complex and diverse systems, distinction between research and care; 5) achieving bal- develop strategies to work across disciplines in new and ance between competing clinical research priorities (e.g. innovative ways, and evaluate the outcomes of collabora- prevention & treatment, domestic & international). Ulti- tive work. Due to the broad range of activities, potential mately, DAIDS seeks to support the goals of the clinical outputs, and outcomes of large research initiatives, it is research networks, individually and collectively, toward essential that a comprehensive conceptual model to serve research progress against HIV/AIDS. as guide for evaluation is developed [17]. Despite some initial work describing the evaluation of science and tech- While most program evaluations are ad hoc, this effort is nology initiatives [6,15,17-21] there is little guidance on unique in that it focuses on creating a system of evalua- methodologies that would be appropriate, and even less tion. The development of an evaluation system for the experience in implementing them [22]. Moreover, while Iterative Communication with Stakeholders Structured Introductions Review/Input Input Informed Evaluation System Plan Preparatory Inquiry Framework Stage Stage Development Stage Process 2 developing an evaluation system Figure for Process for developing an evaluation system. Page 5 of 16 (page number not for citation purposes)
  • 6. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 some have examined the management of clinical trials NIAID and related offices. Within the networks and research [23-25], there is little research on the evaluation DAIDS, individuals representing a wide range of posi- of clinical trials research. tions, job responsibilities and seniority were invited. Many of the challenges found in evaluating scientific The rationale for the involvement of network stakeholders research initiatives can be mitigated by engaging the stake- in the conceptual framework is rooted in participatory holders of the system, particularly those not often approaches to evaluation where those with a stake in the involved in determining organizational evaluation ele- program or initiative (partners, funders, beneficiaries) ments, in the process of evaluation planning. In this sec- assume an active and substantive role across all phases of tion, we describe the approach to engaging stakeholders the evaluation [26,27]. It is believed that participation in in the construction of the conceptual framework to guide identifying relevant evaluation issues and questions, plan- the development of an evaluation system for the DAIDS ning the evaluation design, selecting appropriate meas- clinical research program. ures and data collection methods, and gathering and analyzing data, contribute to enhanced utilization of the To construct the evaluation framework in an informed resultant evaluation [26,28]. At its core, this participatory way, the engagement of a diverse array of stakeholders was evaluation activity provided the opportunity for those deemed necessary. The project sought input from those with close knowledge of and experience with the DAIDS who will ultimately be involved in implementing the eval- clinical research enterprise to proactively design how they uation, including DAIDS staff, Network investigator lead- think the enterprise should be evaluated. This approach ership, the HIV/AIDS Office of Network Coordination, has the dual benefit of yielding evaluative information network evaluation coordinators, community advisory that is useful and meaningful to the people involved in board representatives, and other government agency staff the work, and reduce (or minimize the impact of) the with a role in the networks. Our goal was to achieve a potential imposition of evaluation approaches that are broad sampling of ideas rather than a representative sam- not appropriate to this endeavor or context imposed from pling of persons. Thus, the approach was purposive, sam- outside the initiative. A recent review of a dozen large fed- pling for heterogeneity. The focus on broad eral evaluations revealed that the thoroughness of the heterogeneous participation was intended to ensure con- design process is the most critical factor in successful sideration of a wide variety of view points from those inti- implementation of a large scale evaluation [29]. mately familiar with the networks, provide the necessary breadth and depth needed to accurately describe the proc- Within this context, the key method was a structured esses and outcomes of the clinical research networks, and group conceptualization process (colloquially referred to encourage greater buy-in of the results. as concept mapping) for gathering and organizing input from a wide range of sources. Concept mapping is a well The group of stakeholders to inform this initiative was established mixed methods social research approach that expected to include individuals who: integrates rigorous multivariate statistical analyses with familiar qualitative processes such as brainstorming, sort- • Are familiar with and understand clinical trials net- ing and rating of ideas to create a shared conceptual works from a practice perspective model [30,31]. The concept mapping methodology had been used successfully in similar initiatives [17,18,21,32]. • Have familiarity with government research grant- While a concept map is not necessary for the development making agencies and mechanisms of an evaluation system, the concept mapping methodol- ogy offered several advantages for this development proc- • Have current knowledge of HIV AIDS related clinical ess as it presents a rigorous structured approach that can research be used effectively by scientists, managers, and commu- nity advocates in the process of articulating the conceptual • Understand community perspectives related to HIV and logical models that underlie the complex, collabora- AIDS tive work within the scientific research enterprise. • Have operational and management responsibility The methodology involves several distinct steps. In the for the Division of AIDS first part, the idea generation step, stakeholders were invited to contribute ideas that completed the "focus • Have operational and management responsibility statement" sentence: "Coordinated clinical research net- for network(s) works will be successful if...". Participants represented DAIDS staff and leadership, representatives of other gov- An invitation approach was taken to encourage engage- ernment agencies, networks, clinical trials units and clini- ment at al relevant levels within the networks, DAIDS, cal research sites, the community and other constituencies Page 6 of 16 (page number not for citation purposes)
  • 7. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 having an interest in HIV/AIDS clinical trials research. In clusters that appear small (or 'tight') indicate groups of addition, numerous individual interviews were conducted ideas that were seen as very closely related. and analyzed, together with documents related to the net- works. The statements submitted in the initial idea gener- To facilitate communication about the map and its com- ation were then synthesized into a representative set of ponents, each cluster has been labeled to describe the the- success factors (eliminating duplicates, ensuring clarity matic content of the ideas that comprise the cluster. Just as and relevance to the focus statement, achieving a com- with the individual ideas (numbered points), clusters that mon level of specificity) that participants could work with are physically closer together were deemed more similar, during the idea structuring phase. according to participants, than those more distant from one another. It does not matter whether an idea appears The ideas were then structured [33-35] by having stake- at the top or bottom of the map; a map could be rotated holders sort them into categories of conceptual similarity, in any direction, so long as the relationships between the and by rating them on their importance to the success of points are preserved. Distance between points – concep- the networks, using a 1 (relatively unimportant) to 5 tual similarity – is what matters on a concept map. For (extremely important) scale. more information about concept mapping in the context of this project, please see Concept Systems, Inc., 2008 Using a series of statistical analyses (multidimensional [41]. scaling, then hierarchical cluster analysis), participants' views about how to categorize the success factors were The evaluation framework depicts the broad range of fac- aggregated [33-35]. The result is a single, co-authored con- tors that, in the view of the stakeholders, affect the success ceptual map of the territory that illustrates the relation- of the coordinated clinical research networks. The results ships between ideas [36-38]. The map clusters organize illustrate the breadth of the activities and outcomes asso- the input into groups of ideas (major concepts) [39,40]. ciated with this type of large scale, collaborative scientific Participants' ratings of the ideas make it possible to deter- research initiative. mine priorities, compare the importance of different suc- cess factors, assess consensus across groups, and create Analysis and interpretation of the framework value maps for specific groups of stakeholders. Focusing on each of the clusters within the concept map, and the ideas which comprise them (Additional File 1) Evaluation framework findings allows for a more detailed analysis of the stakeholder-iden- Overview of the framework tified success factors in relation to one another and to the Over 300 stakeholders, representing every sector of the goals and guiding principles for the restructured clinical HIV/AIDS clinical research stakeholder community, con- research networks. Additional File 1 shows the individual tributed ideas that completed the "focus statement" sen- ideas that make up each cluster. It also presents the average tence, "Coordinated clinical research networks will be importance rating (on a 1–5 scale) for each of the ideas and successful if...". More than 1500 ideas were generated in the average importance rating of the statements within each response to that focus statement, which were then synthe- cluster. This table allows us to compare the values that sized into a representative set of 91 ideas that describe stakeholders, as a group, place on broad themes (clusters), what stakeholders believe are the factors critical to the suc- as well as individual ideas. Using the ratings to compare cess of the coordinated clinical research networks. Ninety subgroups of participants to each other to determine simi- stakeholders sorted the ideas by similarity/relatedness, larity or divergence of opinion on the importance of the and 308 stakeholders rated the ideas with respect to their items on the map to evaluation, we compared ratings of importance to the success of the research networks. Table those who identified themselves as related to different sci- 1 summarizes the demographic characteristics of those entific research emphases, (e.g. vaccine research and devel- who completed the ratings. opment, translational research/drug development, optimization of clinical management, and prevention of The concept map representing the success factors is shown HIV infection). Overall, we found a great deal of corre- in Figure 3. On the map, each idea (represented by a point spondence between different stakeholder groups' average labeled with an identifying number), is shown in relation cluster ratings, such that a strong correlation existed from to all of the other ideas in the framework. Ideas that are group to group on the conceptual areas that emerged as closer together were deemed similar (or related) by the being of highest importance and those considered less individuals who sorted them. In contrast, ideas that are important, indicating similarity in the stakeholder-derived farther apart on the map were judged to be less conceptu- patterns across the elements of the evaluation framework. ally similar. The clusters, indicated by the shaded poly- gons, reveal categories of ideas. Larger clusters signify Beginning with the cluster labeled Biomedical Objectives groups of ideas that are more 'loosely' related, whereas (situated at approximately "1 o'clock" on the map), this Page 7 of 16 (page number not for citation purposes)
  • 8. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 Table 1: Demographic characteristics of raters Participant Characteristics Questions Response Choices Frequency % Years Involved 0–5 years 52 17% 6–10 years 60 19% 11–15 years 73 24% 16–20 years 81 26% Longer than 20 years 38 12% did not respond 4 1% 308 100.00% Primary Area of Science Vaccine Research & Development 53 17% Translational Research/Drug Development 38 12% Optimization of Clinical Management, including Co- Morbidities 112 36% Microbicides 17 6% Prevention of Mother- to- Child Transmission of HIV 23 7% Prevention of HIV Infection 23 7% Other/Not applicable 38 12% did not respond 4 1% 308 100.00% Primary Role Network Core 81 26% Clinical Trials Unit/Site 107 35% Laboratory 14 5% Government 47 15% Community 12 4% Pharma/Biotech 2 1% Pharmacy 2 1% Institutional Review Board/Ethics Committee 0 0% Non- Governmental Organization 10 3% Advisory Group/Committee 6 2% Other 23 7% did not respond 4 1% 308 100.00% Page 8 of 16 (page number not for citation purposes)
  • 9. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 group of ideas focuses on the ultimate aim of the trials One of the greatest challenges to a cooperative group is to networks to identify prevention and treatment strategies make choices from among an array of competing ideas. In for HIV/AIDS that can lead to fewer new infections, and striving to achieve balance in their research agendas, net- reduced morbidity and mortality in people living with works are constantly weighing factors such as portfolio HIV. Of the eight clusters on the map, Biomedical Objec- diversity, stakeholder input, scientific opportunity, capa- tives emerged as the most important success factor. That bility and feasibility. Interestingly, both this Scientific cluster had the highest average importance score (4.11) Agenda Setting cluster, and the Biomedical Objectives cluster (Additional File 1) and it includes the highest rated connect very strongly to one of the most important guid- (across all the 91 success factors in the framework) indi- ing principles for the restructured networks, namely, vidual statement, "produce high-quality, scientifically responsiveness (Section III A.). That these two clusters valid results" (#82 with a rating of 4.74). Traditional peer were both rated among the top 3 success factors (average review usually places the significance of biomedical importance of 4.06), and were seen as highly related (as research objectives highest, in terms of importance, when indicated by proximity on the map) would appear to evaluating research proposals in the biomedical sciences. affirm a close alignment between DAIDS goals in restruc- That the biomedical objectives of the clinical research net- turing the networks, and what the network stakeholders works were seen, across stakeholder groups, as the most believe most important for success. important component of success suggests that, at least in this context, large research initiatives do not differ from Next (again moving clockwise) is a large cluster labeled conventional investigator-initiated research. Collaboration, Communication and Harmonization. The emphasis of the ideas in this cluster is, as its name implies, Moving "clockwise" on the map, the adjacent cluster, Sci- on collaboration and communication both within and entific Agenda Setting, is comprised largely of statements between networks. Statements in this cluster (Additional that reflect the need for, and the means by which, net- File 1) reference collaboration activities at the scientific works identify their research priorities (Additional File 1). level, e.g. "the vision and goals are shared" (#55), as well Figure 3 Concept map of success factors Concept map of success factors. Page 9 of 16 (page number not for citation purposes)
  • 10. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 as at the operational level, e.g. harmonizing key functions The cluster entitled Resource Utilization contains subtopics like training, laboratory, and data management across that, taken together, describe the concept of resources in networks. The contents of this cluster reflect the restructur- several ways, including human, time, materials and loca- ing principles of efficiency and coordination, which tions. For example, human resources issues include that emphasize an integrated approach to HIV/AIDS vaccine, "principal investigators demonstrate scientific leadership prevention and therapeutics clinical trials, in order to and innovation" and that they are able to "commit ade- maximize research opportunities and respond to public quate time to network activities." Specific attributes of health needs in an increasingly efficient manner. The staff are mentioned, including the need for diversity and importance of linkages across the scientific priority areas expertise in infrastructure development. Other statements was a key element in the planning process for the restruc- focus more on issues related to clinical research site capac- turing and was the primary impetus for issuing a single ity and presumably refer to the many HIV/AIDS trials sites network leadership RFA (as opposed to separate RFAs for located in developing countries, where access to resources each clinical research area, as had been done in the past). may be limited. Thus, a common theme across both the This was also the major impetus for the establishment of people and the functions is related to making wise use of the Office of HIV/AIDS network coordination, formation resources. This set of statements relates closely to the prin- of a network Principal Investigators committee, the ciple of capacity building that figured prominently in the assembly of external scientific advisory bodies to assist network restructuring design. The principles of efficiency with cross-network scientific prioritization, and strength- and coordination also permeate this cluster. Capacity at ened coordination across NIH Institutes and Centers. the site level, in particular, is a focus of existing network Interestingly, in contrast to the previous clusters, this evaluation activities. Resource Utilization is the fifth most group of ideas was viewed as being among the lowest in important cluster (out of 8), with an average rating of importance among the success factors, with an average 4.00. rating of 3.81 (keeping in mind that, in this framework, least important does not mean unimportant). Given that On the left side of the map, Community Involvement many of the statements in the cluster are more process-ori- emerged as one of the four highest rated success elements ented, it is not surprising that participants view the longer in the framework at 4.05. Within this cluster, the idea with term, ultimate goals of the network (such as in the Biomed- the highest importance rating is: "the dignity and human ical Objectives and Scientific Agenda Setting clusters) as rights of participants are respected." (#42). In addition to more important than the activities that lead to these types the protection of human subjects, the statements in this of outcomes. Also, the activities of resource sharing and cluster emphasize the importance of community involve- inter-network coordination are largely a new demand on ment at all levels and in every stage of study development the networks, mandated through the restructuring proc- and implementation. Adequate training and support ess. While stakeholders recognize this concept as a factor must also be provided to enable this community involve- in the success of the networks, these results would suggest ment. This cluster sheds light on a topic that may be of that it is not yet fully valued by all participants. special significance to clinical trials (by comparison to other types of research). Community involvement plays Similarly, Operations and Management is also rated rela- an essential role in ensuring the ethical and scientific tively lower in importance (3.80) by participants. Again, quality of HIV/AIDS clinical research, its relevance to this cluster pertains to administrative, rather than scien- affected communities, and its acceptance by such commu- tific aspects of the networks' activities. The statements in nities (2007 UNAIDS document Ethical considerations in this cluster also indicate that operations and management HIV preventive vaccine research). Community involvement depend upon both the networks and DAIDS and suggest may also be more challenging with HIV/AIDS (vs. other that a comprehensive evaluation of the enterprise diseases) because the communities of interest often reside includes both the grantees and the funding agency, in resource poor settings (internationally and domesti- DAIDS. cally), the social stigmas attached to HIV/AIDS, and the "disparities in power, wealth, education and literacy Attention to funder issues is even more explicit in the which often exist between the individuals proposing to DAIDS Policies and Procedures cluster. As stated by stake- conduct research and those who are hardest hit by the HIV holders, for the coordinated clinical research networks to epidemic" [42]. This cluster relates directly to one of the reach their goals, DAIDS policies must "reflect what is key components of the network restructuring – the new required for good science, protection of human subjects, Community Partners organization, whose very purpose is and safety", be "transparent", "streamlined" and "clear". to enhance community input on all levels, and assure This cluster is very much process oriented, and is rated rel- effective representation of and timely communication atively lower in importance compared to other clusters among the many communities involved with network (3.96). research. Page 10 of 16 (page number not for citation purposes)
  • 11. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 Closely related to the issue of community involvement is framework, allowing the logic model to represent the Relevance to Participants. The focus of this cluster is on hypothesized causal pathways between elements identi- ensuring that network research: a) addresses questions of fied through the concept mapping process. The current relevance to the communities and trial participants, and; logic model for the DAIDS clinical research networks is b) can produce results leading to preventions and treat- depicted in Figure 4. ments that can be expected to be made available to the study participants and their communities. Special popula- Typically, logic models are constructed from lists of tions affected by the HIV/AIDS epidemics raise significant inputs, activities, outputs, and outcomes, often developed ethical considerations, some of which are captured in this and considered categorically and independent form one cluster. This cluster's location directly adjacent to Biomed- another. The spatial features found in the concept map are ical Objectives (signifying that stakeholders found these important because it suggests that those clusters of ideas elements to be closely related), is also notable, as this clus- that are closer are more likely to be connected in the logic ter addresses issues regarding who is involved in the stud- model. The understanding of how these concepts are ies and who will be the beneficiaries of the results. related facilitates a more informed and efficient transition Relevance to Participants emerged as a very important suc- from the map to the logic model. For example, the loca- cess factor (4.09), second only to the Biomedical Objectives tion of the DAIDS Policies and Procedures cluster in relation of the networks. to the Biomedical Objectives cluster suggests they assume opposite ends of the logic model. The contextual knowledge required to understand or derive value from the concept map can vary depending on Each box in the logic model corresponds to a component the user's focus. For those whose interest is to gain a or cluster obtained from the concept mapping analysis. sophisticated understanding of what the stakeholders of The logic model describes a presumed program theory for the HIV/AIDS trials networks believe are the factors criti- the complex research initiative and conveys the sequence cal to the success of the venture, it would be important to of expected processes and outcomes. In the end, the logic have: 1) a good familiarity with the NIH's and the investi- model developed as a part of this approach was based on gators' goals for the networks; 2) knowledge of the diver- a set of concepts that came directly from the stakeholders; sity and multidisciplinarity within the networks; 3) a with the components of the final logic model directly background in how clinical trials are conducted; 4) a grasp linked to the original concept mapping ideas submitted on the global pandemic from many aspects including the by the stakeholders. Thus, the logic model is more repre- epidemiology, demographics, scientific and ethical chal- sentative of the reality of the research networks, than had lenges; 5) an understanding of the special operational and only one component of the system developed the model. logistical challenges of HIV/AIDS clinical research; and 6) This collaboratively developed model serves as a guide for knowledge of the environment/milieu of related/sur- identifying critical outcome pathways of greatest rele- rounding efforts in which the networks function. With vance and for identifying where measurement should be this kind of contextual knowledge, the relationships focused as the plan is implemented. between individual statements (points) and groups of statements (clusters) on the map would have find their An Evaluation Measurement Task Force (EMTF), consist- greatest meaning and could potentially provide the most ing of evaluation coordinators from each of the clinical insight as to how the data could be utilized. trials networks, along with selected DAIDS staff with eval- uation expertise and knowledge of the networks, assisted From evaluation framework to evaluation plan in interpreting the framework and logic model. The EMTF The concept map of success factors provides a framework used the concept map of success factors and the logic upon which subsequent evaluation planning is continu- model as the foundation upon which to develop propos- ously built. The concept map was used as the basis for a als for evaluation questions (what should be evaluated) logic model of the coordinated clinical research networks and measures (how those priority elements should be that captures stakeholders' assumptions about how pro- evaluated), as well as how these efforts should be opera- gram activities lead to desired outcomes [32,43]. tionalized in the networks and DAIDS. The EMTF's oper- Together, the concept map and logic model form an inte- ationalizing efforts focused on compiling evaluation grated view that captures the complexity of the research tools, processes, and methods from networks, developing enterprise as well as how it could be evaluated. The con- potential indicators for concepts found in the framework, cept mapping analysis and results describe the elements of and identify potential data sources. Thus, the involvement success and their relative importance, and the logic model of the EMTF in further interpretation and planning sequences the elements of success temporally and caus- ensured a direct correspondence between the map, con- ally. Graphically, the concept map depicts the multivari- tent, questions, and measures. This linkage is important as ate related components of the emerging evaluation data is collected and analyzed in the future. Page 11 of 16 (page number not for citation purposes)
  • 12. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 Inputs Activities Outputs Short Term Longer Term Outcomes Outcomes/ Impact Recruitment & Community retention goals met Involvement (Community Involvement) Community input Scientific Agenda Scientific agenda of Relevance to High quality Treatment and (Community Setting processes Participants scientific results prevention Involvement) and increased measures knowledge (Biomedical Increased capacity of (Biomedical Objectives) Collaboration & developing world sites Objectives) Communication (Resource Utilization ) Appropriate human and (Communication, Collaboration, Integrated use of infrastructure Harmonization) developing world sites Resources are (Resource Utilization) Results Published in place (Biomedical HIV/AIDS Scientific research Objectives) mortality and : morbidity plan and priorities Operations and reduced (Scientific Agenda Management (Biomedical Funder’s Setting) processes in Objectives) Policies and place Procedures Harmonized systems and procedures (Communication, Objectives Collaboration, Harmonization) *Bolded text indicates corresponding concept map cluster Figure 4 DAIDS clinical trials networks logic model DAIDS clinical trials networks logic model. The evaluation system design approach being described model and associated potential measures, tools and herein, seeks to incorporate a set of measures that, when resources and recommendations for piloting an initial taken together, can meet the varied needs of system stake- cross-network evaluation system. With this type of design holders and yield results from which reasonable inferences in place, evolving iterations of the system can be tested for about performance, across the entire clinical trials enter- feasibility and utility, and modified as needed. prise, can be made, including DAIDS, the trials networks, community, clinical research support services, etc. In some Often, evaluations of research programs are done as one areas on the concept map (e.g. Operations and Manage- time, ad hoc events, with little precedent for systems ment) evaluation metrics that pertain to well-established approaches. (See for exceptions, the National Science functions and processes, will likely include measures previ- Foundation's Engineering Research Centers [16] and the ously developed, utilized and refined within individual tri- National Cancer Institute's Transdisciplinary Tobacco Use als networks, and DAIDS. Reuse of existing measures offers Research Centers (TTURCs) initiative [18,21]). In con- the potential to build on past experience, gain efficiency, trast, this effort takes a systems approach to evaluation, establish baselines and, possibly, performance bench- seeking to build rigorous evaluation into the funding marks. In other areas (e.g. cross-network coordination and process and lifecycle, so it is seamlessly integrated with collaboration) there may be little experience to draw upon, other research activities and provides feedback at key in which case new measures may need to be explored and points to inform decision-making at all levels. Thus, the tested for utility and feasibility. final evaluation plan, when developed, will describe sev- eral key events and milestones that need to be addressed The outcome of this participatory process will be a draft in a comprehensive evaluation system. This approach also evaluation plan that describes the framework and logic recognizes that evaluation questions, resources and activ- Page 12 of 16 (page number not for citation purposes)
  • 13. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 ities can change depending upon the stage of the initia- biomedical objectives, how those objectives are vetted, tive, demands upon scientific study, and many other how they align (or not) with those of other complemen- operational and management factors, and as such, can tary efforts, how well they match (or not) the strengths inform a flexible evaluation system capable of responding and resources of the networks, and how successful have to the kinds of shifts that take place in a research environ- the networks been in reaching these objectives. A similar ment as dynamic is this one. examination can occur across all clusters of success factors yielding a great deal of detailed information about what Conclusion should be measured and how. Additionally, the findings The trend toward "big science" calls for the development allow us to see what other 'clusters' of success factors are of systems for managing and evaluating large scale, multi- seen by the stakeholders as being related, both conceptu- site, collaborative scientific research initiatives. The need ally and in their importance. For example, the findings for evaluation systems is particularly great given increas- reveal that scientific agenda setting (including prioritiza- ing pressure for accountability for public program out- tion) and relevance to study participants are both seen as comes. Yet, large scale science has its own unique goals, being closely related to the biomedical objectives and context and requirements that must be taken into account nearly as important. This provides additional layers of when designing ways to evaluate success. An ideal scien- understanding and again, suggests potential directions in tific research enterprise has been defined as one that 1) which the evaluation system development can proceed, in invests in work that impacts significant social and scien- direct alignment with what the networks themselves see as tific challenges and responds to new discoveries; 2) fosters the most important activities for achieving success. a wide network of relationships that generates relevant questions, recognizes emerging issues, and sustains signif- The findings from this process allow us to map, in even icant, cutting-edge programs of work; 3) develops and greater detail, the success factors at multiple levels (i.e. nurtures the human and organizational capacity to con- statement or cluster) to the demographics of the stake- duct research; and 4) recognizes and communicates its holder participants. For instance, we can start to deter- impact on the world [44]. To date, there a few examples to mine if within one area of science (e.g. HIV vaccines) draw upon, which indicate how best to model the success- certain success factors are seen as more important than ful evaluation of this type of large scale research initiative. they are viewed by stakeholders working in a different In this context, DAIDS' newly restructured clinical area, such as HIV therapy. Similarly, we can learn how research networks are an example of a large research initi- stakeholders' time in the networks or their roles relates to ative seeking to proactively address multiple challenges as their views of what is important for network success. This it adapts to meet the ideals of a successful research enter- kind of information can help to reveal the basis of conflict prise. or competition which could be creating obstacles (e.g. within networks, between networks or network collabora- At the highest level, the findings from this process provide tors) and point to activities which could be undertaken to a 'global' view and insight into the network stakeholders' mitigate such. thoughts, views, and perceptions of the factors critical to the networks' success. The HIV/AIDS clinical research net- Taken together, these findings and the processes described works comprise thousands of individuals with a variety of here are the first of their kind relative to this large research skills and training, working in hundreds of different initiative and provide valuable insights into the nature organizations, in almost 50 different countries. The diver- and functioning of the HIV/AIDS clinical research net- sity of expertise, interests, priorities and cultures repre- works. The concept map and the logic model reflect eval- sented in the networks, while vital to their strength, pose uation criteria that account for the goals of multiple challenges to maintaining a sense of unity of purpose and stakeholder groups and inform the first efforts to develop teamwork. The findings reveal areas of the network that a systematic approach to evaluating this complex enter- stakeholders see as mission critical, and the relative prise. importance of these different areas. For example, we have learned, for the first time, that across the board, stakehold- In this paper, we described a stakeholder-driven concep- ers believe that the biomedical objectives selected by the tual framework development process as an initial step in networks, are the most important factor for network suc- the creation of an evaluation system. This participatory cess. This finding alone indicates that despite the diversity activity engaged scientists, staff, and community advo- inherent in the networks, science is the primary driver. cates with first-hand experience of the initiative, in co- Given our interest in developing a comprehensive constructing the framework for an evaluation system. This approach to the evaluation of the networks, this finding stakeholder-driven process served two purposes. It was provides a data-driven basis for taking next steps, such as designed primarily as a prospective planning tool to iden- efforts to understand how the networks determine their tify the success factors of the coordinated clinical research Page 13 of 16 (page number not for citation purposes)
  • 14. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 networks and, therefore, elements of the evaluation sys- emerging database of "big science" evaluation planning tem. It also enabled a "pulse check" on the extent to which efforts where concept mapping was used to conceptualize stakeholders have embraced the goals and guiding princi- elements for evaluation. As this collection of conceptuali- ples embedded in the guiding principles behind the recent zation projects increases, the potential for meta-analysis restructuring of the clinical trials networks. across participants, ideas, data (sorting and rating), and findings (maps and patterns) is possible. Moreover, we detail how the results of the concept map relate to the goals of the networks, and point to how the Several limitations and caveats of this work are worth not- map can potentially be used to guide the development of ing. First, the results of the development of a conceptual evaluation plans for this complex initiative. That said, the framework are context specific. Although several clusters map can inform other 'views' or interests, for which an that emerged from the concept mapping process were extensive background in HIV/AIDS clinical research similar to the content found in other large scale research would not be required. For example, for someone inter- evaluation planning processes where concept mapping ested in researching the dynamics and complexities of has been used, the ideas generated, organized and priori- team science, a contextual knowledge of that field could tized by the stakeholders were specific to the DAIDS clin- inform the interpretation of the HIV/AIDS network con- ical networks. Thus, while limiting the generalizability of cept map (presented herein) in a very different way, and the results, the use of concept mapping had the advantage for a different purpose. In a sense, it depends on the 'lens' of focusing the scope of the evaluation, increasing the through which the user wishes to view the map, as this potential for an efficient and sustained evaluation effort. conceptualization activity could support multiple inquir- Second, the results of the development of the conceptual ies. framework for the evaluation are highly dependent upon informed and meaningful participation. While we sought In the context of evaluation of large scale scientific to engage broad-based participation from those that have research initiatives we found support for the idea that "big a stake in the processes and outcomes of the network science" is accountable for a wider variety of processes, activities, it is clear that some groups had greater numbers activities and outcomes than traditionally organized sci- of participants than others. This is fairly typical in partici- ence research. This finding is consistent with other writing patory evaluation planning. Concept mapping has several on this subject [3,5,19,20,45]. It is not sufficient to simply advantages for the management of participation, includ- apply the same processes of evaluation traditionally asso- ing multiple opportunities for engagement, multiple ciated with individual investigator awards because the forms of input, and multiple products for interpretation. expectations of large scale science are broader. Given the complexity, size, and scope of the activities of We also found that the sheer size and complexity of the the clinical research networks, the time spent considering initiative requires attention to a host of management chal- and engaging stakeholders in the conceptualization proc- lenges that would not be applicable to traditional individ- ess is likely to yield benefits in later stages of the evalua- ual investigator awards. The recent restructuring of the tion. Additionally, we used mean cluster ratings to coordinated clinical research networks was undertaken to represent the perspectives of the participants. This address many of these management issues, focusing on approach means that inferences about ratings must be efficiency, coordination, responsiveness, capacity build- made with caution. The primary intent of this project was ing and evaluation. This assessment indicated an aware- to sample for heterogeneity to assure that a varied range of ness of these management issues among stakeholders, but people had an opportunity to volunteer to participate and revealed that overall, these stakeholders place more value that no one who might want to take part would be on outcomes than the administrative and management excluded. While average ratings may represent well the processes required to achieve those outcomes. views of the participants themselves and of the best repre- sented subgroups of participants, it is not possible to say Finally, we found a number of ideas and concepts that how well such averages represent groups that did not vol- emerged from the concept map that were consistent with unteer at comparable rates. In general, sampling issues in other evaluation planning efforts carried out in large sci- this context are a challenge and the opportunistic volun- entific research initiatives. Concepts related to scientific teer sampling used here was intended as a compromise outputs and goals, communication and collaboration, between inclusion and representativeness. If subgroup and management policies and procedures have appeared comparisons are desired, we will incorporate a represent- in other framework development efforts [17,21,32]. Sim- ative sampling approach. ilarities notwithstanding, the conceptualization process was unique to the DAIDS clinical research networks. Through this example, we've described an approach to a Nonetheless, we believe that this work contributes to an participatory conceptual framework development process Page 14 of 16 (page number not for citation purposes)
  • 15. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 as a part of evaluating a large scale clinical research 3. Nash SJ, Stillman BW: Large-scale biomedical science: Explor- ing strategies for future research. Washington, DC, USA: The endeavor. These approaches to planning and identifying National Academies Press; 2003. key elements of an evaluation system may serve as useful 4. Institute of Medicine: NIH extramural center programs. Wash- guidance to others who are also pioneering in this emerg- ington, DC, USA: The National Academies Press; 2004. 5. Inventory and Evaluation of Clinical Research Networks: Core and ing field. descriptive survey final report 2006 [http://www.clinicalresearchnet works.org/documents/CDESum.pdf]. 6. Manning FG, McGeary M, Estabrook R, Committee for Assessment of Competing interests NIH Centers of Excellence Programs, Institute of Medicine, Board on The authors declare they have no competing interests. Health Sciences Policy: National institute of health extramural center pro- grams: criteria for initiation and evaluation 2004 [http://www.nap.edu/ catalog.php?record_id=10919#orgs]. Authors' contributions 7. Smith R: Measuring the social impact of research: difficult but All of the authors helped to conceptualize ideas, interpret necessary. Br Med J 2001, 323:528. findings, and review drafts of the manuscript. JK con- 8. Brainard J: New science measures related by OMB. Chron High Educ 2002, 48:. ceived of the study and drafted significant portions of the 9. Brainard J: White House proposes criteria for measuring manuscript. MK contributed to the project design and progress and results of federally financed research. Chron High Educ 2002, 48:. interpretation of the results and provided project support 10. U.S. General Accounting Office: NIH research: Improvements needed in and management at each step. KQ conducted the concept monitoring external grants. Washington, D.C 2000 [http://www.gao.gov/ mapping analysis, the background literature review and new.items/h100139.pdf]. 11. U.S. Congress. Legislature: National Institutes of Health Reform Act of contributed to the overall manuscript writing. SR assisted 2006. Washington, D.C 2005 [http://www.govtrack.us/congress/bill in revisions of the manuscript. WT provided guidance to text.xpd?bill=h109-6164]. the background research and project implementation. All 12. Brodsky R: Bush orders agencies to appoint 'performance improvement officers'. Government Executive.com 2007 [http:// authors approved the final manuscript. www.govexec.com/dailyfed/1107/111407ts1.htm]. 13. Bush GW: Executive order: improving government program performance. The White House 2007 [http://www.whitehouse.gov/ Additional material news/releases/2007/11/20071113-9.html#]. 14. National Research Council: Evaluating federal research programs: research and the government performance and results act 1999 [http:// Additional file 1 www.nap.edu/catalog.php?record_id=6416]. Washington, D.C.: Statements organized by cluster with average importance ratings by National Academies Press 15. National Academy of Sciences, National Academy of Engineering, & statement and cluster. Individual statements within concept map clus- Institute of Medicine: Implementing the government performance and ters, with mean importance ratings, by statement and cluster. results act for research, a status report 2001 [http://www.nap.edu/cata Click here for file log/10106.html#toc]. Washington, D.C.: National Academies Press [http://www.biomedcentral.com/content/supplementary/1478- 16. Committee on Facilitating Interdisciplinary Research, National Acad- 4505-7-12-S1.doc] emy of Sciences, National Academy of Engineering, Institute of Medi- cine, Committee on Science, Engineering, and Public Policy: Facilitating interdisciplinary research 2004 [http://www.nap.edu/cata log.php?record_id=11153#orgs]. Washington, D.C.: National Acade- mies Press 17. Trochim WM, Markus SE, Masse LC, Moser RP, Weld PC: The eval- Acknowledgements uation of large research initiatives: a participatory integra- This project has been funded in whole or in part with funds from the United tive mixed-methods approach. Am J Eval 2008, 29(1):8-28. States Government Department of Health and Human Services, National 18. Stokols D, Fuqua J, Gress J, Harvey R, Phillips K, Baezconde-Garbanati Institutes of Health, National Institute of Allergy and Infectious Diseases, L, Unger J, Palmer P, Clark M, Colby S, Morgan G, Trochim W: Eval- uating transdisciplinary science. Nicotine Tob Res 2003, under subcontracts to Concept Systems, Inc. through an Interagency 5(Suppl):S21-S39. Agreement between the National Institute of Allergy and Infectious Dis- 19. National Academy of Sciences, National Academy of Engineering, & eases and U.S. Military Human Immunodeficiency Virus (HIV) Research Institute of Medicine: An assessment of the national science foundation's Program (Cooperative Agreement No. W81XWH-04-2-005) and contract science and technology centers program 1996 [http://www.nap.edu/ books/0309053240/html/]. Washington, D.C.: National Academies N01-AI-50022 ("HIV Clinical Research Support Services"). The authors Press wish to thank Daniel Montoya for leadership on this project during the 20. Committee on Science, Engineering, and Public Policy, National Acad- planning phases; Melissa G. Burns for managing data collection during the emy of Sciences, National Academy of Engineering, Institute of Medi- concept mapping phase and consultant team project support; Brendan Cole cine: Experiments in international benchmarking of US research fields 2000 [http://www.nap.edu/catalog.php?record_id=9784]. Washing- for project support; and Gruschenka Mojica for assistance in manuscript ton D.C.: National Academies Press preparation. The authors also wish to thank the staff of the HIV/AIDS 21. Quinlan KM, Kane M, Trochim WM: Evaluation of large Research Office of Network Coordination, HIV/AIDS network researchers and staff, initiatives: outcomes, challenges and methodological consid- Division of AIDS program staff, the Evaluation Measurement Task Force erations. New Directions for Evaluation 2008, 118:61-72. 22. O'Hara J, Araujo JJ, Choi M, Pepper C, Wagner R, Wang G, Woollery and all others who contributed to the project. T: Evaluating Public Research Investment. Presentation at the Annual Conference of the American Evaluation Association, Baltimore, MD, References USA 2007. 1. Hanney SR, Gonzalez-Block MA: Building health research sys- 23. Dilts DM, Sandler AB, Baker M, Cheng SK, George SL, Karas KS, tems to achieve better health. Health Res Policy Syst 2006, 4:10. McGuire S, Menon GS, Reusch J, Sawyer D, Scoggins M, Wu A, Zhou 2. Edgerton DEH: Before big science: the pursuit of modern K, Schilsky RL: Processes to activate phase III clinical trials in a chemistry and physics, 1800–1940. Ann Sci 1999, 56(1):107-107. cooperative oncology group: the case of cancer and leuke- mia group b. J Clin Oncol 2006, 24(28):4553-4557. Page 15 of 16 (page number not for citation purposes)
  • 16. Health Research Policy and Systems 2009, 7:12 http://www.health-policy-systems.com/content/7/1/12 24. Dilts DM, Sandler AB: The "invisible" barriers to clinical trials: the impact of structural, infrastructural, and procedural bar- riers to opening oncology clinical trials. J Clin Oncol 2006, 24(28):4545-4552. 25. Doroshow JH: CTWG evaluation plan results of baseline fea- sibility analysis. National Cancer Institute 3rd meeting of the clinical tri- als advisory committee. Bethesda, USA 2007. 26. Cousins JB, Whitmore E: Framing participatory evaluation. New Directions for Evaluation 1998, 80:5-23. 27. King JA: Making sense of participatory evaluation practice. New Directions for Evaluation 1998, 80:57-67. 28. Patton MQ: Utilization-focused evaluation 3rd edition. Thousand Oaks, CA, USA: Sage Publications; 1997. 29. Howell EM, Yemane A: An assessment of evaluation designs, case studies of twelve large federal evaluations. Am J Eval 2006, 27(2):219-236. 30. Trochim WM: An introduction to concept mapping for plan- ning and evaluation. Evaluation and Program Planning 1989, 12:1-16. 31. Kane M, Trochim WM: Concept mapping for planning and evaluation Thousand Oaks, CA, USA: Sage Publications; 2007. 32. Andersen L, Gwaltney M, Sundra D, Brownson R, Kane M, Cross A, Mack R, Schwartz R, Sims T, White C: Using Concept Mapping to Develop a Logic Model for the Prevention Research Centers Program. Prev Chronic Dis 2006, 3(1):. 33. Coxon APM: Sorting data: collection and analysis Thousand Oaks, CA, USA: Sage Publications; 1999. 34. Rosenberg S, Kim MP: The method of sorting as a data gather- ing procedure in multivariate research. Multivariate Behav Res 1975, 10:489-502. 35. Weller SC, Romney AK: Systematic Data Collection Newbury Park, CA, USA: Sage Publications; 1988. 36. Davison ML: Multidimensional Scaling New York, NY, USA: John Wiley and Sons; 1983. 37. Kruskal JB, Wish M: Multidimensional scaling Beverly Hills, CA, USA: Sage Publications; 1978. 38. Shepard RN, Romney AK, Nerlove SB: Multidimensional scaling: theory and applications in the behavioral sciences Volume 1. New York, NY, USA: Seminar Press; 1972. 39. Anderberg MR: Cluster analysis for applications New York, NY, USA: Academic Press; 1973. 40. Everitt B: Cluster Analysis 2nd edition. New York, NY, USA: Halsted Press, A Division of John Wiley and Sons; 1980. 41. Concept Systems, Inc [http://www.conceptsystems.com/down load/DAIDS_Methodology_FAQ_02-26-06_Kagan_Quinlan.pdf] 42. Joint United Nations Programme on HIV/AIDS (UNAIDS): Good par- ticipatory practice guidelines for biomedical HIV prevention trials. France 2007 [http://data.unaids.org/pub/Manual/2007/ jc1364_good_participatory_guidelines_en.pdf]. 43. W.K. Kellogg Foundation: Using logic models to bring together planning, evaluation and action: logic model development guide. Michigan 2004 [http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf]. 44. Pardo TA, Dawes SS, Cresswell AM, Thompson F, Kumar Tayi G: Finding our future: A research agenda for the research enterprise Albany, NY, USA: Center for Technology in Government, University at Albany, SUNY; 2002. 45. Inventory and Evaluation of Clinical Research Networks: Best practices study final report. Rockville 2006 [http://www.clinicalresearchnet works.org/documents/BPESum.pdf]. Publish with Bio Med Central and every scientist can read your work free of charge "BioMed Central will be the most significant development for disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK Your research papers will be: available free of charge to the entire biomedical community peer reviewed and published immediately upon acceptance cited in PubMed and archived on PubMed Central yours — you keep the copyright Submit your manuscript here: BioMedcentral http://www.biomedcentral.com/info/publishing_adv.asp Page 16 of 16 (page number not for citation purposes)