art-cross-eval

PLEASE SCROLL DOWN FOR ARTICLE
This article was downloaded by: [DEFF]
On: 25 June 2009
Access details: Access Details: [subscription number 789685088]
Publisher Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,
37-41 Mortimer Street, London W1T 3JH, UK
Social Epistemology
Publication details, including instructions for authors and subscription information:
http://www.informaworld.com/smpp/title~content=t713765921
Quality Issues in Cross-disciplinary Research: Towards a Two-pronged
Approach to Evaluation
Jens Aagaard-Hansen; Uno Svedin
Online Publication Date: 01 April 2009
To cite this Article Aagaard-Hansen, Jens and Svedin, Uno(2009)'Quality Issues in Cross-disciplinary Research: Towards a Two-
pronged Approach to Evaluation',Social Epistemology,23:2,165 — 176
To link to this Article: DOI: 10.1080/02691720902992323
URL: http://dx.doi.org/10.1080/02691720902992323
Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf
This article may be used for research, teaching and private study purposes. Any substantial or
systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or
distribution in any form to anyone is expressly forbidden.
The publisher does not give any warranty express or implied or make any representation that the contents
will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses
should be independently verified with primary sources. The publisher shall not be liable for any loss,
actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly
or indirectly in connection with or arising out of the use of this material.

Social Epistemology
Vol. 23, No. 2, April–June 2009, pp. 165–176
ISSN 0269–1728 (print)/ISSN 1464–5297 (online) © 2009 Taylor & Francis
DOI: 10.1080/02691720902992323
Quality Issues in Cross-disciplinary
Research: Towards a Two-pronged
Approach to Evaluation
Jens Aagaard-Hansen and Uno SvedinTaylor and FrancisTSEP_A_399404.sgm10.1080/02691720902992323Social Epistemology0950-0693 (print)/1464-5289 (online)Original Article2009Taylor & Francis232000000April-June 2009JensAagaard-Hansenjah@life.ku.dk
In recent decades there has been increasing demand for and considerable efforts to conduct
cross-disciplinary research. However, assessment of research quality in such endeavours
still is often based on mono-disciplinary criteria and not seldom carried out by reviewers
without strong cross-disciplinary experience. The authors suggest a two-pronged approach
to cross-disciplinary research evaluation. One part should comprise an individual review
of all the disciplines involved based on their mono-disciplinary sets of criteria. The other
part should be a separate evaluation of the cross-disciplinary aspects based on the review of
“problem formulation”, “integration and scope of the disciplines”, “parts and the whole”,
“practical managerial aspects” and “the applied aspects” (when relevant). The pros and
cons of implementing this approach in a stepwise manner or simultaneously is discussed. It
is suggested that funding agencies develop more fair sets of review procedures for cross-
disciplinary research and show willingness to allocate extra funds and time to such forms
of research that sometimes are regarded as relatively more “risky” than conventional
mono-disciplinary types.
Keywords: Criteria; Cross-disciplinarity; Evaluation; Inter-disciplinarity; Multi-
disciplinarity; Research; Trans-disciplinarity
Jens Aagaard-Hansen is a senior researcher with a double training background as medical doctor and anthropol-
ogist. For the past 16 years he has been based at DBL – Centre for Health Research and Development, Denmark.
He has been involved in cross-disciplinary research, applied medical anthropology and research capacity strength-
ening mainly in developing countries. Uno Svedin, Ph.D. in physics from Stockholm University 1974 and later
Associate Professor in Physics, broadened his scope to systems analysis, futures studies, natural resources and
environmental policy. From 1979 he served at the Swedish Council for Planning and Co-ordination of Research
(FRN), the last years up to 2000 as Director of Research. In 2001 he was appointed Director of International Affairs
at the then newly created Swedish Research Council for Environment, Agricultural Sciences and Spatial Planning
(Formas) and has part time been Professor at Tema, Linköping University 1993–2004. He is also currently part-
time Senior Research Fellow at the Resilience Center, Stockholm University. Correspondence to: Jens Aagaard-
Hansen, DBL – Centre for Health Research and Development, Faculty of Life Sciences, University of Copenhagen,
Thorvaldsensvej 57, DK-1871 Frederiksberg C, Denmark. Email: jah@life.ku.dk
DownloadedBy:[DEFF]At:12:2325June2009

166 J. Aagaard-Hansen and U. Svedin
Introduction
Although most research still is mono-disciplinary in character, research involving
more than one discipline has become increasingly frequent. This has been the case both
for research in applied domains where a societal or technical problem has been in
focus, and in very basic, “curiosity-driven” research at the borderline between “old”
disciplines. In some cases these cooperative ventures have stabilised themselves as
“new” disciplines in the academic portfolio. This has opened a lot of new possibilities
and new, innovative research questions have been formulated and addressed.
Those who ventured into cross-disciplinary research1
have had to harmonise old
meansandfindnewformsofdatacollectionmethods,dataanalysis,studydesign,ethical
codes of conduct and new ways to construct theoretical frameworks based on different
paradigms (Aagaard-Hansen 2007; Aagaard-Hansen and Johansen 2008; Aagaard-
Hansen, Johansen, and Riis 2004; Olsson and Sjostedt 2004; Svedin and Hägerhäll
Aniasson 2002). Researchers have overcome these new challenges with varying degrees
of success depending on the areas of application as well as the skills of its practitioners.
However, the intellectual and institutional systems for most research activities are
still rooted in a mono-disciplinary environment. For a long time, most such institu-
tions of higher education have been organised according to disciplinary domains—
often within distinct boundaries, although there is a clear tendency today of a
dynamic change over time. The traditional “peer review” provides the golden stan-
dard of scientific evaluation. “The competitive nature of academic life is a function of
the emphasis placed on gaining a professional reputation …” (Becher and Trowler
2001, 118) “… and the peers are the judges” (Becher and Trowler 2001, 86). However,
the fairness of the practice (although not necessarily the principle) of peer reviews has
been questioned when it comes to cross-disciplinary activities (Epton, Payne, and
Pearson 1983, 44; Porter and Rossini 1985). Although their focus was on evaluation of
institutions rather than projects, Hultberg et al. phrase it like this: “Traditional
systems of evaluation will only, we argue, reproduce old knowledge structures and
make changes harder” (1998, 171).
Assessment of cross-disciplinary research should, thus, be based on appropriate
criteria. As expressed by Berge and Powell:
a crucial difference between basic mono-disciplinary research on the one hand and inter-
disciplinary research on the other hand, is that the former finds its legitimisation within its
own field. In this sense disciplines are bodies of knowledge or objects to which it is possible,
even respectable, to add something. Inter-disciplinary research has no mechanism of
intrinsic legitimisation and rather depends upon how well it illuminates the overarching
problem being researched. (1997, 17–18)
The authors continue:
In addition to the basic criteria used to evaluate research in general there is a need for
elaborated criteria specifically for the judgement of inter-disciplinary research within
academia, where problem-solving is not enough. In particular, criteria employed to assess
the relevance of the research will probably differ from mono- to inter-disciplinary works.
(Berge and Powell 1997, 22; see also Friedman and Friedman 1985, 81)

Social Epistemology 167
The increase in the number of cross-disciplinary research projects has recently been
followed by a large number of publications exploring various theoretical aspects of
such works, but a review is beyond the scope of the present article. However, there has
only been limited attention to the particular aspects of evaluation (for example, Janssen
and Goldsworthy 1996; Wickson, Carew, and Russell 2006), and they focus mostly on
the ex post perspective (i.e. the outcomes after the research has been conducted) rather
than ex ante (i.e. the proposal assessment phase).
The present article aims at exploring various aspects of quality of cross-disciplinary
research. Are there criteria that are common for all sorts of research irrespective of
whether they are mono-disciplinary or cross-disciplinary? Are there some additional
criteria that apply only to cross-disciplinary research? Do different kinds of cross-
disciplinary research call for different criteria? How should the review processes be
conducted (in terms of criteria, qualification of reviewers and the procedures) in
order to evaluate cross-disciplinary research in a fair way? It is our hope that a clarifi-
cation of these issues will be of help to funding agencies of various sorts and for the
criteria they apply for research review procedures as well as to researchers who plan to
conduct cross-disciplinary research and to academic bodies responsible for the struc-
ture of research.
Cross-disciplinary Research: Clarifying the Terminology
King and Brownell point out that the concept of discipline is a complex phenomenon
with social as well as cognitive aspects, “a community, a network of communications,
a tradition, a particular set of values and beliefs, a domain, a mode of enquiry, and a
conceptual structure” (1966, quoted in McNeill 1999, 316). This definition will guide
this article too. Cross-disciplinarity can be characterised by two main variables: “inte-
gration” and “scope”. The level of integration of the different disciplines is expressed
in the taxonomy of cross-disciplinary research, which was outlined by Rosenfield
(1992, 1351) in her seminal article and which will be used in this article:
Level one: Multidisciplinary. Researchers work in parallel or sequentially from disciplinary-
specific base to address a common problem. The total result of the research effort appears
as the sum of the partial efforts with a low level of further integration.
Level two: Interdisciplinary. Researchers work jointly but still from disciplinary-specific
basis in interactive modes of operation in order to address a common problem. Integration
efforts are given care and interest but not to the extent that the “input” competences have
lost their specificities.
Level three: Transdisciplinary. Researchers work jointly using shared conceptual frame-
works that are specifically designed for the purpose of a particular research endeavour and
drawing together disciplinary-specific theories, concepts, and approaches to address a
common problem.
The general perception of, and distinction between, multi-disciplinarity and inter-
disciplinarity is shared by the vast majority of relevant scholars (for example, Albrecht,
Higginbotham, and Freeman 2001, 73; Epton, Payne, and Pearson 1983, 4–5; Friedman

and Friedman 1985, 77; Gilbert 1998, 6; Kilburn 1990, 132; Kline 1995, 2; Maina-
Ahlberg, Nordberg, and Tomson 1997, 1230; McNeill 1999, 314). Although they
approach it in different ways and may even have slightly different views on key points,
there is a general agreement that inter-disciplinarity entails more integration than
multi-disciplinarity (for more detailed review, see Choi and Bak 2006). In this article
the term “cross-disciplinarity” will be used in the way it has earlier been used by OECD
(1972) and Rosenfield (1992); that is, as a general designation for all the three terms
(multi-disciplinarity, inter-disciplinarity, and trans-disciplinarity). (See also OECD
1998.)
However, not only the degree of intended integration but also “the academic
distance” (the “scope”) between the involved disciplines plays a role. As McNeill
explains: “the nature of both bridge-building and restructuring will differ according to
what I call the ‘scope’ that is, the number of disciplines involved and the ‘distance’
between them” (1999, 314). Berge and Powell state that:
some disciplines, though belonging to the same faculty, rarely seemed to collaborate or to
collaborate well; anthropology and sociology being cases in point. On the other hand,
anthropologists collaborated well with vegetation biologists and economists with popula-
tion biologists. The explanation that was put forth …, was that certain disciplines, although
belonging to very different traditions, shared basic ideas about what was important or not.
Field biologists and anthropologists met in regarding observations or data locally obtained
as central to their research traditions, whereas economists and population biologists shared
an interest in modelling. (1997, 19)
One of the main justifications of cross-disciplinarity is said to be its relevance to
pertinent problems (Berge and Powell 1997, 6; Higginbotham, Albrecht, and Connor
2001, xxii; McNeill, García-Godos, and Gjerdåker 2001, 3). Although drawn from
experiences gathered from industrial Research & Development (R&D), Epton, Payne,
and Pearson give a clear definition: “By the term ‘applied’ we mean R&D carried out to
achieve an externally imposed and tangible technological objective, as distinct from
basic research aimed at extending or deepening scientific knowledge” (1983, 12). With
a link to her taxonomy of cross-disciplinary health research, Rosenfield (1992, 1353)
proposes that there is a direct link between the level of disciplinary integration and the
contribution to health policies and programmes—a tenet that is supported by Berge
and Powell (1997, 17). It should be emphasised, however, that justifications for needs
of cross-disciplinary approaches may be found in both “curiosity-driven” types of
research and in those where an externally set target in terms of a question to be solved
is involved (Svedin 1991; Svedin et al. 1999).
What is Quality in Research?
Are there any common quality criteria that apply to mono-disciplinary as well as
cross-disciplinary research? We believe there are. According to Berge and Powell
(1997, 20–21) the main points are:
● explicit and well founded problem-formulation;
● well-defined terminology;

● consistency;
● demonstration of knowledge of relevant literature;
● operationalisation—arguing for the kind of data that are needed to illuminate the
problem; and
● methodology that is arguably capable of procuring the data.
The study should be well focused. It should be consistent, in the sense that all parts of
the proposal should be natural elements of the whole and that there should be no gaps
(e.g. the general objective should be equivalent to the sum of the specific objectives).
The proposed research question should be in harmony with other known relevant
research (i.e. it should not simply duplicate previous research, but provide an appro-
priate, innovative “next step”). The proposed project should be realistic in the sense
that the resources (manpower, time and funds) should be sufficient for the task. The
researchers should have the necessary “disciplinary” competences to conduct the
various aspects of the study. The various key issues (e.g. design and key concepts)
should be sufficiently clear. The practical aspects and the often widely neglected issue
of “research management” should also be considered (Aagaard-Hansen and Ouma
2002).
Apart from these universal virtues of proposals there are some criteria that are disci-
pline specific. For instance, anthropology (and several other disciplines) emphasises a
sound description of the theoretical framework as an integrated part.
Some funding agencies emphasise the outcome of a strengthening of research
capacity as an important criterion in its own right. This argument is equally relevant
for mono-disciplinary as for cross-disciplinary projects.
Which Additional Criteria are Necessary in Ex Ante Evaluation
of Cross-disciplinary Research?
On top of the universal criteria outlined above, we suggest a set of five (partly inter-
related) ex ante evaluation criteria of particular relevance to cross-disciplinarity:
The Problem Formulation
This is a key issue. Are there indications that a cross-disciplinary approach is more
suitable for addressing the research question than a mono-disciplinary approach? Has
the innovativeness of the research question been appropriately matched by the cross-
disciplinary approaches applied? Does the project explicitly indicate the source from
which the research question comes (e.g. a response to issues of relevance to society)?
The Integration and the Scope of the Disciplines
Based on the problem formulation, which disciplines are participating? Are there good
arguments for the presence of all the disciplines? Is a significant one missing? Is the
cross-disciplinary approach de-focusing the project or could it be considered that the

approach on the contrary does contribute to focus? To which extent have the research-
ers in the project integrated their conceptual framework, concepts and approaches
(Rosenfield 1992, 1351) and is the result of the intended integration in harmony with
the research questions? How ambitious is the project in terms of integration (i.e. how
strong is the move towards trans-disciplinarity)? The more integration, the stricter the
evaluation should be with regard to the “integration criteria”.
The Parts and the Whole
Given the disciplines that are included in the proposal, how are the parts integrated
into a whole? This point comprises the terminology; that is, the way concepts are
defined and used in the proposal—an aspect that is particularly important when differ-
ent disciplines bring together different “traditions”. Three different situations may be
distinguished. In the first case, a concept (term) is derived from one discipline and is
not used in any other (e.g. “heteroskedacity”). Secondly, there may be concepts that
originate in one discipline, but have been adapted by others (e.g. “resilience”). This
opens the possibility for different interpretations. Thirdly, there may be ill-defined
terms that are generally used (e.g. “sustainable”). Also the integration of methods is an
issue here. This is a question of harmonising design and methods of data collection and
analysis. Are there indications that the team members possess the necessary, mutual
knowledge of each others’ disciplines?
Applied Aspects
In many research councils this is termed “relevance”. Cross-disciplinary research is
generally more dedicated to provide “relevant” results. However, there is a need to
define relevant for what purpose and for which type of actors or sectors. Berge and
Powell recommend that the applied aspect should come in more strongly in cross-
disciplinary research assessments (1997, 22). The emphasis on applicability as a
variable in the evaluation process is also commented on by Rostrup-Nielsen (1997, 28)
and McNeill, García-Godos, and Gjerdåker (2001, 37). However, as was stated above,
some cross-disciplinary research is “curiosity driven” and therefore has to be handled
differently with regard to the “applied criteria”.
The Cross-disciplinary Competence of the Research Team
This is primarily a question of whether the necessary cross-disciplinary expertise is
present. Does the team seen as a whole include members that can competently conduct
all of the activities outlined? Are the skills available for the level of integration aimed
for? The project proposal should indicate how the benefits of the cross-disciplinarity
approach should be operationalised; for example, through joint workshops for
planning, fieldwork and write-ups. Furthermore, the competence of the managerial
leadership is very important. Epton, Payton, and Pearson emphasise that “management
and process variables must be given added weight” (1983, 43) in cross-disciplinary

research evaluation. Thus, research management criteria, which are important for
mono-disciplinary efforts, are even more crucial for cross-disciplinary research.
The Ex Post Evaluation Perspective
Above, the evaluation process has been discussed as far as ex ante reviews of proposals
are concerned, and this is also our key focus. However, as a comparison we still want
to touch on some particular issues related to ex post evaluation of a whole cross-
disciplinary project where a number of additional issues come into play. The assessment
should be based on the same general quality standards that apply to scientific merit; that
is, cross-disciplinary projects should not be treated more leniently. At the same time we
claim that cross-disciplinary research is more difficult. Thus, when it comes to the
assessment of publication rates and similar metric judgements, it could be argued that
extra weight could be given to cross-disciplinary publications in respected scientific
journals as the time needed to achieve a convergence in a cross-disciplinary endeavour
may be longer than in a more narrow, mono-disciplinary project where the method-
ological issues are already at hand. Hence there should be a sort of “bonus” given to
cross-disciplinary researchers for each larger project well carried through, given of
course that assessments in terms of the peer-reviewed quality of the project is good both
in parts and in its integration. A criterion of project success with regard to cross-
disciplinarity could also be the publication of joint publications—see, for instance,
Aagaard-Hansen and Ouma (2002), who recommend that the number of articles co-
authored by researchers from more than one discipline can be a quantitative indicator.
Apart from the rather straightforward counting of articles and monographs, there
are more soft aspects that are nevertheless important outcomes of cross-disciplinarity
(as well as mono-disciplinary) projects. Good research is characterised by raising just
as many questions as answers. Consequently, an evaluation criterion could be to look
for cases where a project has directly resulted in (innovative) new research questions
and maybe subsequent spin-off projects.
Another important aspect of ex post evaluation relates to the reflection over the path
taken by the project, including changes of assumptions, stumbling blocks perceived
and overcome, and redirections of efforts in general, that is the process itself. The
relationship here to what happened to initial ideas of integration is of course of strong
interest. Instead of assessing changes from the initial design as negative points, a well-
understood process of redesign facing the experiences of the process may be seen as
something positive in the overall judgement. This applies to both mono-disciplinary
and cross-disciplinary activities, but is of more profound importance for the cross-
disciplinary cases as they normally have to design their paths less from a standard chart
than in more mono-disciplinary approaches.
Operating at Different Levels: Evaluation of Institutions and Programmes
The discussion above focused on the criteria and processes of importance to the evalu-
ation of individual, cross-disciplinary projects. However, at times evaluations are

conducted at “higher levels”. For instance, whole research institutions may be subject
to evaluation, in which case there is a need to look not only at the various affiliated
projects, but also at the departments, staff profiles, incentive structures (formal and
informal), research priorities, sources of funding, synergy between various institu-
tional resource bases and collaborative links to other research groups.
At an even higher level, international research collaborations are sometimes exam-
ined to see whether their strategic research initiatives are working according to the
intentions. For instance, issues like the harmony between overall objectives related to
the policy of a certain programme and de facto projects being funded may be of key
interest. Also comparisons between different internationally competing institutions are
important with regard to their respective roles, international distribution of labour and
specific areas of key competence. Sometimes the capacity of an institution to be
assessed to extract funding from a wide set of sources could also be used as an indicator.
Bordons et al. (1999) as well as Thi and Lahatte (2003) and Jeannin and Devillard
(2005) provide examples of evaluation of cross-disciplinarity conducted at institu-
tional programmatic levels.
Towards an Improved Evaluation of Cross-disciplinary Research:
A “Two-pronged Approach”
As stated above, we contend that the present evaluation procedures often are likely to
be biased against cross-disciplinary research unless specific measures are taken institu-
tionally, procedurally and in terms of assessment criteria. In order to pay justice to
cross-disciplinarity, we suggest that a two-pronged review procedure be adapted and
further developed with regard to cross-disciplinary research proposals.
● Part 1:
Evaluation of the contributing individual, mono-disciplinary elements is done
accordingtotheirownstandardsinatraditional,disciplinarypeer-reviewjudgement.
Assessment of the expertise of the researchers to implement the proposed research
with regard to the individual disciplines is done in the same setting. The researchers
involved in the project should be required to show evidence of a threshold level of
competence in at least one discipline relevant to the consortium.
The managerial and practical aspects (including budgeting) should be adequate.
This is an important and often neglected aspect of proposal evaluation.
● Part 2:
This part of the evaluation focuses on the specific cross-disciplinary aspects. It refers
to “the problem formulation”, “the integration and the scope of the disciplines” and
“the parts and the whole” as described above.
Both of these aspects of evaluation are mandatory components. However, there can be
different time perspectives. In some cases there can be a sequence where part 1 precedes
part 2 and serves as a screening; that is, only when the proposal lives up to the quality

criteria for each of the disciplines involved, should the proposal proceed to the second
step where the cross-disciplinary aspects are assessed. The proposal should be
supported only if both of these steps are satisfactory. Alternatively, the two parts are
assessed at the same time in a concurrent approach. This allows for situations where a
proposal may be supported based on overall high merit in spite of weakness in one (or
more) of the involved disciplines. We find that both approaches are acceptable depend-
ing on the circumstances. The step-wise approach is stricter in a procedural sense and
perhaps sets higher standards for the “parts”, but is also more time consuming and
“rigid”. The concurrent approach is more flexible, but potentially also more “lax”
depending on how it is implemented. We are not adamantly in favour of one or the
other as long as there are no compromises regarding the overall quality of the cross-
disciplinary project and with special regard to the integration aspects outlined above.
As already stated, management is important for all research, but it is even more
important in cross-disciplinary projects and should be given specific attention. Issues
such as the leadership’s understanding of collaborative processes and the way the
project’s total vision is framed and carried by the responsible participants are also of
importance. Normally the qualifications of the principal investigator are essential.
Alternatively, research teams without previous experience can involve facilitators with
the necessary background.
The issue of “relevance” of the research will depend on the extent to which the
individual project but also the research funder (e.g. a research council) will choose to
emphasise this aspect, but generally “relevance” will be more prominent as an assess-
ment criterion in cross-disciplinary research.
We find it crucial that reviewers who are given the evaluation responsibility should
have some kind of personal cross-disciplinary background in addition to their disci-
plinary skills—either in terms of double training or in terms of practical involvement
in similar projects. This point is supported by McNeill, García-Godos, and Gjerdåker,
who point out it is important that reviewers “should have proven competence not
simply in one of the appropriate disciplines, but in interdisciplinary research as such”
(2001, 33–34). The key issue here is to search for additional, cross-disciplinary compe-
tences over and above required disciplinary competences.
McNeill, García-Godos, and Gjerdåker call cross-disciplinary research “a high risk/
high return endeavour” (2001, 8). The concept of “risk-taking” is a necessary element
of nurturing new avenues of research. At the same time there may be ways of minimis-
ing the risk and still achieving innovative and highly rewarding results. The implication
is that the funders of research should allocate more funds and allow for a longer time
horizon than when mono-disciplinary research is considered (see also Roussel, Saad,
and Erickson 1991, 78).
Conclusions and Recommendations
Neither researchers nor funding agencies are value-free. They reflect the surrounding
societal environment and the prevailing ideologies and trends. This is expressed in their
organisational structure as well as in their set of explicit and implicit status and power

relations and related assessment criteria. At present, cross-disciplinarity is generally not
held in sufficiently high esteem within the traditional institutions of higher education.
Forfurtherelaborationofthistheme,seeforinstancetheanalysisinTheNewProduction
of Knowledge (Gibbons et al. 1994; see also later publications by these authors). In the
context of this article the aim is more limited and relates to the need for developing a
fairer and more appropriate evaluation process for cross-disciplinary research. In this
article we have ventured an outline of a two-pronged approach and a set of evaluation
criteriathataremoreappropriateforevaluationofcross-disciplinaryresearchproposals
than some of those used today. As one part of the assessment, the various disciplines of
the project should be considered individually with regard to scientific merit, expertise
of the researchers and managerial aspects. As another and equally important part, cross-
disciplinary research should be assessed regarding specific aspects of problem
formulation, integration between the disciplines, cross-disciplinary expertise of the
researchersandtheleadershipcompetencewithsuchprojects.Itistheevaluationofboth
of these parts that should lead to the final decision when cross-disciplinary projects
compete. It is optional whether these two parts are applied sequentially or concurrently.
Apart from the importance of agreeing on a set of suitable criteria, it is also necessary
for the research councils to ensure appropriate competences of the review panels.
Funding bodies should also be encouraged to consider how the element of risk taking
should be prioritised and how the allocation of sufficient time and funds for selected
projects can be assured.
Such evaluation processes are relevant for ex post as well as ex ante situations and
apply not only to individual projects, but also at higher institutional and programme
levels. Although we are mainly addressing the research council perspective, we believe
that parallel considerations should be made by the researchers and institutions of
higher education. The suggestions raised in this article may also have some implica-
tions for journals publishing cross-disciplinary articles.
We hope that the present article will contribute to the further development and
upgrading of the quality of evaluations of cross-disciplinary research and thereby
promote the standards of cross-disciplinary research.
Acknowledgements
This article is based on the merged experiences of the two authors accumulated in
different settings. The authors are profoundly thankful to the many colleagues and
collaborators from whom they have learned a lot over the years. Special thanks go to
Professor Desmond McNeill for many important suggestions and input to the text.
Positions in this paper are only provided in a personal capacity and do not necessarily
reflect the positions of the various institutions where the authors work or have worked.
Note
1
[1] We use this term to include multi-disciplinary, inter-disciplinary, and trans-disciplinary
research—see below.

References
Aagaard-Hansen, J. 2007. The challenges of cross-disciplinary research. Social Epistemology 21 (4):
425–38.
Aagaard-Hansen, J., and M. V. Johansen. 2008. Research ethics across disciplines. Anthropology
Today 24 (2): 15–19.
Aagaard-Hansen,J.,M.V.Johansen,andP.Riis.2004.Researchethicalchallengesincross-disciplinary
andcross-culturalhealthresearch:thediversityofcodes.DanishMedicalBulletin51(1):117–20.
Aagaard-Hansen, J., and J. H. Ouma. 2002. Managing interdisciplinary health research—Theoretical
andpracticalaspects.InternationalJournalofHealthPlanningandManagement17(3):195–212.
Albrecht, G., N. Higginbotham, and S. Freeman. 2001. Transdisciplinary thinking in health social
science research: Definition, rationale, and procedures. In Health social science. A transdisci-
plinary and complexity perspective, edited by N. Higginbotham, G. Albrecht, and L. Connor,
pp. 70–89. Oxford: Oxford University Press.
Becher, T., and P. R. Trowler. 2001. Academic tribes and territories. Buckingham: The Society for
Research into Higher Education and Open University Press.
Berge, G., and N. Powell. 1997. Reflections on interdisciplinary research. A synthesis of experiences
from research in development and the environment. Working Paper 1997:4, Centre for
Development and the Environment (SUM), University of Oslo, Oslo.
Bordons, M., M. A. Zulueta, F. Romero, and S. Barrigón. 1999. Measuring interdisciplinary
collaboration within a university: The effects of the multidisciplinary research programme.
Scientometrics 46 (3): 383–98.
Choi, B. C. K., and A. W. P. Pak. 2006. Multidisciplinarity, interdisciplinarity and transdisciplinarity
in health research, services, education and policy: 1. Definitions, objectives, and evidence of
effectiveness. Clinical and Investigative Medicine 29 (6): 351–64.
Epton, S. R., R. L. Payne, and A. W. Pearson. 1983. Introduction. In Managing interdisciplinary
research, edited by S. R. Epton, R. L. Payne, and A. W. Pearson, pp. 1–45. Chichester: John
Wiley & Sons.
Friedman, R. S., and R. C. Friedman. 1985. Organized research units in academe revisited. In
Managing high technology: An interdisciplinary perspective, edited by B. W. Mar, W. T. Newell,
and B. O. Saxberg, pp. 75–91. Amsterdam: Elsevier Science Publishers B.V., North-Holland.
Gibbons, M., C. Limoges, H. Nowotny, S. Schwartzman, P. Scott, and M. Trow. 1994. The new
production of knowledge. London: Sage.
Gilbert, L. E. 1998. Disciplinary breadth and interdisciplinary knowledge production. Knowledge,
Technology, and Policy 11 (1&2): 4–15.
Higginbotham, N., G. Albrecht, and L. Connor. 2001. Introduction to Health social science. A trans-
disciplinary and complexity perspective, edited by N. Higginbotham, G. Albrecht, and L.
Connor, pp. xxi–xxvi. Oxford: Oxford University Press.
Hultberg,J.,C.Rosenberg,S.Thorpenberg,L.Nordholm,A.Elzinga,P-O.Brogren,andB.Samuelsson.
1998. A model for the study of research and education in a transdisciplinary context.Knowledge,
Technology, and Policy 11 (1&2): 167–90.
Janssen, W., and P. Goldsworthy. 1996. Multidisciplinary research for natural resource management:
Conceptual and practical implications. Agricultural Systems 51 (3): 259–79.
Jeannin, P., and J. Devillard. 2005. Implementing relevant disciplinary evaluations in the social
sciences. National vs international interactions in scientific communities. Scientometrics 63
(1): 121–44.
Kilburn, K. D. 1990. Creating and maintaining an effective interdisciplinary research team. R&D
Management 20 (2): 131–8.
Kline, S. J. 1995. Conceptual foundations for multidisciplinary thinking. Stanford, CA: Stanford
University Press.
Maina-Ahlberg, B., E. Nordberg, and G. Tomson. 1997. North–South research collaboration:
Challenges in institutional interaction. Social Science and Medicine 44 (8): 1229–38.

McNeill, D. 1999. On Interdisciplinary Research: With particular reference to the field of environment
and development. Higher Education Quarterly 53 (4): 312–32.
McNeill, D., J. García-Godos, and A. Gjerdåker, eds. 2001. Interdisciplinary research on development
and the environment. SUM Report No. 10, Centre for Development and the Environment,
University of Oslo, Oslo.
OECD. 1972. Interdisciplinarity: Problems of teaching and research in universities. Paris: OECD.
——. 1998. Interdisciplinarity in science and technology. DSTI/STP(98)16, OECD, Paris.
Olsson, M-O., and G. Sjostedt, eds. 2004. Systems approaches and their application—Examples from
Sweden. Dordrecht: Kluwer Academic Publishers.
Porter, A. L., and F. A. Rossini. 1985. Forty interdisciplinary research projects: Multiple skills and
peer review. In Managing high technology: An interdisciplinary perspective, edited by B. W.
Mar, W. T. Newell, and B. O. Saxberg, pp. 103–12. Amsterdam: Elsevier Science Publishers
B.V., North-Holland.
Rosenfield, P. L. 1992. The potential of transdisciplinary research for sustaining and extending
linkages between the health and social sciences. Social Science and Medicine 35 (11): 1343–57.
Rostrup-Nielsen, J. 1997. Hvordan bliver forskning dynamisk? [How does research become
dynamic?]. In Den vanskelige balance—en bog orn forskningsledelse [The difficult balance—a
book about research management], edited by Akademiet for de tekniske Videnskaber, pp. 25–
50. Vaerloese, Denmark: Akademiet for de tekniske Videnskaber.
Roussel, P. A., K. N. Saad, and T. J. Erickson. 1991. Third generation R&D. Managing the link to
corporate strategy. Boston, MA: Harvard Business School Press.
Svedin, U. 1991. Transdisciplinary approaches to environmental issues. From natural science to
social science and the humanities—Towards a contextual understanding. Paper presented at
the International Symposium on Interdisciplinarity, UNESCO, Paris, 16–19 April.
Svedin, U., A.-L. Lindén, D. Magnusson, O. Stendahl, G. Tibell, D. Vågerö, G. Öqvist, and J. Larsson.
1999. Tvärvetenskap—hur, av vem och varför? [Cross-disciplinarity—how, by whom and
why?]. Report from the Joint Swedish Research Councils Expert Committee on Cross-
disciplinarity, Swedish Council for Planning and Co-ordination of Research (FRN),
Stockholm, September, 1–27.
Svedin, U., and B. Hägerhäll Aniasson, eds. 2002. Sustainability, local democracy and the future: The
Swedish model. Dordrecht: Kluwer Academic Publishers.
Thi,T.U.N.,andA.Lahatte.2003.Measuringandassessingrelativedisciplinaryopennessinuniversity
research units. Research Evaluation 12 (1): 29–37.
Wickson, F., A. L. Carew, and A. W. Russell. 2006. Transdisciplinary research: Characteristics,
quandaries and quality. Futures 38: 1046–59.

art-cross-eval

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Viewers also liked

Viewers also liked (6)

Similar to art-cross-eval

Similar to art-cross-eval (20)

More from Jens Aagaard-Hansen

More from Jens Aagaard-Hansen (10)

art-cross-eval