Applying Performance Management Principles To A Learning Environment For Undergraduate Dissertations A Case Study

Applying Performance Management Principles to a Learning
Environment for Undergraduate Dissertations: a Case Study
Taken from The International Journal for Quality and Standards Page 1 of 23
www.bsieducation.org/ijqs Dr. Jacqueline Day & Dr. Milena Bobeva
Applying Performance Management Principles to a Learning Environment for
Undergraduate Dissertations: a Case Study
Dr. Jacqueline Day and Dr. Milena Bobeva
The Business School
The Institute of Business and Law
Bournemouth University
Poole
BH12 5BB
ABSTRACT
In the UK an individual research work, often known as a ‘dissertation’, is a common
requirement of undergraduate and taught postgraduate degrees in business, law and
humanities subjects. However, changes in the business environment have placed heavy
demands of dissertation students, their supervisors and academic departments. Amongst
these challenges are increasing cohort sizes, individual supervision mode, coordination of
many stakeholders and increasingly stringent quality regulations to be met within tight
financial and resource budgets.
This research presents a response to these problems, by demonstrating how one Higher
Education institution developed a Learning Environment (LE) specifically designed to manage
dissertations. Implementation of this LE has lead, over several years, to apparent
improvements to the student dissertation experience and achievement and also seemed to
have enhanced the consistency and effectiveness of research supervisors. However, to justify
further investment of time and money and to further develop operational management it
became necessary to properly assess the performance and impact of the LE. The case study
discussed in the paper explains how the performance management regime was designed and
then examines how it will be used in the longer term to nurture a community of practice for all
involved in dissertations.
Keywords: student, dissertation, learning environment, performance management

1. Introduction
In commerce and industry business performance management (BPM) is a well accepted
discipline that enables strategy to be executed in ways to meet organisational objectives
(Becher, 2005; Clayton, 2005). However, the implementation of BPM within the Not-for-Profit
organisations such as local government, health services or education is not so well
understood (Rees and Gardner,2003). With respect to the latter, case studies and exemplars
are rare and the limited research that has been carried out in the area of performance
management with in Higher Education Institutions (HEIs) has been directed to executive or
policy-making strata (Taylor, 2001) rather than at ‘nuts and bolts’ level of the organisation.
The conventional view is that evaluation enables targets to be set, against which the work of
individuals or groups will be judged and so will positively influence the behaviour of individuals
or functions to meet these targets (Athiyaman, 1997). At the institutional level, performance
indicators for Higher Education (HE), particularly those that involve a final aggregation in the
form of league tables, have received a mixed response. However, the particular focus of the
project was upon the internal settings, which by their nature are rarely published. In the
absence of theoretical guidance, the pragmatic view endorsed by Taylor (2001, p.392) was
used to guide design of performance management by judging the salience of measures for
the problem domain by:
‘…the extent to which they improve the quality of the decisions made by the people who are
exposed to them, whether this is the government, university management or the individual
academic.’
Accordingly, the purpose of the present work is to address this lacuna in current knowledge
by providing an ideographic study that reports experiences with applying BPM principles to
the development and utilisation of an integrated learning environment (LE) to support
undergraduate dissertations within UK Business School. These ideas have been tested in
practice as they present one of the major challenges to the management of academic studies.
The paper begins with a discussion of the nature of dissertations and the problems
associated with these. Further it presents how these problems have been addressed in the
form of a LE and introduced the phases of its development, grounded in the philosophy of
BPM.

1.1 What are Dissertations?
An individual piece of work demonstrating understanding of the research process and critical
understanding of a substantive knowledge domain is usually called a ‘dissertation’. This type of
studies often commands a prominent position as the ‘flagship’ unit of most business,
information systems, social science, humanities or law degrees and their completion is deemed
by many institutions to be the element demonstrating the ‘honours worthiness’ of the award. In
the realm of student-led research, academic prestige, funding and institutional pride are
traditionally grounded in the demands of doctoral level studies (and for that matter taught
Master’s dissertations) but in terms of the number, anecdotal evidence suggests that these are
dwarfed by undergraduate studies. Despite this, undergraduate level research activities have
attracted comparatively little academic attention. This neglect is surprising, considering the
commitment bound up in this exercise and its implication for the development of those generic
skills prized by industry and commerce. Furthermore, the dissertation being (usually) the most
rewarding part of an undergraduate degree course (Ramsden, 1992) student experience of the
research can be a factor in encouraging the individual to subsequently enrol for a higher
degree.
1.2 Challenges for the Student
What comes over particularly clearly from the existing studies is that it is at the undergraduate
level where the change to the learning context for the student is perhaps at its most acute.
Since it is here the budding researcher must adopt a more intense level of engagement with
their chosen specialist subject area (Clauston and Whitcombe, 2005; Rowley, 2004). Indeed,
the student finds her/himself having to deal with a set of profound and often uncomfortable
transitions (Table I):
From To
Interaction within the institution-----------------------------------------Æ External communication
General knowledge--------------------------------------------------------Æ Knowledge in-depth
Knowledge consumer-----------------------------------------------------Æ Knowledge provider
Single truth ------------------------------------------------------------------Æ Multiple/provisional truth
Structured time allocation-------------------------------------------------Æ Self management of time
Delivery to a student group-----------------------------------------------Æ One-to-one with supervisor
Directed----------------------------------------------------------------------- Æ Autonomous learning
Pupil----------------------------------------------------------------------------Æ Academic collaborator
Table I: The Impact of the Dissertation Upon Undergraduate Student
In these matters as Todd et al (2006 p.171) rightly observe:

‘while for the motivated and able student the dissertation can be an intellectually stimulating
and rewarding experience which is a marker of student ability, for those who are weaker or
less motivated, it can be an unpleasant and unsettling for both student and supervisor.’
Inability to cope with these changes, combined with natural doubts and uncertainties
associated with any form of research are likely to have a negative impact upon student
performance such that at its most extreme can cause failure of the degree (Gordan, 2003).
1.3 Challenges for Educational Institutions.
The limited studies of sub-doctoral research carried out to date have been directed toward
specific pedagogical issues and problems. These include topics, such as improving the
students knowledge and application of research methods (Booth and Harrington, 2003; Benson
and Blackman, 2003; Edwards and Thatcher, 2004), analysis of student-supervisor relationship
(Mackinnison, 2004) and assessment regimes (Hand and Clewes, 2000; Milton and Lyons,
2003; Webster et al, 2000). What has not really been looked at is a LE for dissertations that
enables students to deliver improved quality outcomes reflecting higher levels of expectations
about their learning experience and degree grades. Increased exposure to the business world
and the introduction of fees in UK HE have fostered an attitude whereby students increasingly
consider themselves as ‘clients’ of an educational service (Armstrong, 2003; Browne et al,
1998) and demand higher levels of value-for-money from the ‘provider’ (the academic team).
This generates tension with the intention of the academic research exercise, as the novice
researcher must be supported and encouraged to learn to deal independently with the
problems associated with interpreting the literature, setting research objectives through
development of problem-solving skills and a style of knowledge acquisition congruent with the
precepts of self-directed learning (Mackinnon, 2004).
Academics and administrative staff are also not immune from those forces that require the
overseeing of more students by fewer supervisors. Furthermore, taught degree dissertations
now come in a greater variety of forms including the more traditional empirical investigations,
action research, consultancy projects or marketing investigations. These are all legitimate
interpretations of what constitutes undergraduate research and which impact upon the
supervisor–student relationship that lies at the heart of a successful dissertation. Therefore, it
is clear unless an improved level of support is given, what is demanded may well prove
impossible without damage to the supervisor’s own scholarly activity and research agendas
(Doring, 2002).
In this setting, the LE was made the locus of the effort to improve the performance of the
dissertation management. It combines new organisational processes, information technology

and pedagogical practices designed to deliver benefit to students, academic staff,
administrators and the managers of a UK University Business School.
The overarching vision for the LE was to develop something that supports the strategic
objectives of the institution, including the need to improve the student learning experience and
to better realise the research potential of the academic staff. In this regard the development of
the learning environment addresses a paradoxical position of the university: it had few systems
at operational level supportive of its prime mission to research and educate.
2 The General Design of the LE
2.1 The Existing Situation
A range of sources were used to gather knowledge about the then current problems and
where the future stress points were likely to occur (Appendix A). Anecdotal evidence was
obtained from formal and informal feedback from supervisors and students. Documentary
evidence included statistical analysis of marks and the results from investigations of formal
student complaints and hearings dealing with academic offences. Comments from external
examiners were also scrutinised and discussion groups employed to canvass the views of
course managers and administrators. The findings from the theoretical review and the
empirical evidence were integrated into a set of aspirations (soft objectives) for the future LE
(Table II):

Student Aspirations Institutional Aspirations
Sustain a high level of motivation
for students
Interaction between students and academic staff within
a ‘community of scholars’
Good previous experience of
research, such as dissertations,
assignments and case studies.
The existence of a visible and accepted organisational
strategy for undergraduate units to build up a
knowledge base of investigations done by students.
Willingness to seek advice and
guidance
Availability of a group of lecturers as a shared resource
A supportive and nurturing
relationship with the supervisor
Good fit between the supervisor’s personal
interpretation of the dissertations and the organisation.
A degree of self-awareness of
learning styles
Provision of lecturer-led seminars using a panel of
domain experts to guide the development of the
conceptual framework.
Self-organisation and methodical
approach to matters such as
documentation.
The degree of flexibility in the organisational
understanding of what constitutes a dissertation
A positive attitude toward critical
inquiry in relation to the topic
Knowledge of the supervision arrangements and
process
A willingness to engage in informal
peer assessment
The active use of code of practice for defining
expectations and responsibilities of the parties.
Ability to self-direct their learning
based on feedback and guidelines
More accurate and transparent feedback and
assessment
Awareness of the purposes and
limitations of different
communication channels
Ability to use consistently different communication
channels
A realistic understanding of time-
scheduling and workloads
An accessible and responsive supervisor team, able to
manage effectively students expectations
Table II: General Requirements for Dissertations LE
2.2 The Design of the LE
The requirements suggested that the design of the LE follows a ‘tight-loose’ form alluded to
by Todd et al (2004). Prudent use of information technologies (Alavai et al, 1997; Clerehan et
al, 2003) address the requirements of the parties delivered through improved work flow, the
monitoring of support for dissertations, enhancement assessment regimes, better
communication, as well as providing easy access and visualisation of relevant documents,
such as official teaching and learning policy statements and guidelines. Ideally, the new LE
would be able to rapidly adapt to the changing agendas of the university. However, it was
recognised this need would be difficult to implement, not least because high level institutional
strategies are often vague and frequently contradictory (Rowley, 2000). There is a whole set
of constraints upon the design of any programme level management environment that
originates in the general environment that impacts on the efficacy of learning environments
(Aldridge and Rowley, 1998; Oldfield and Baron, 2000). For example, the usefulness of a

book shop or the adequacy of university library stock, are outside the remit of either students
or supervisors, at least in the short term. There was little that could be done about these
factors but to try to take them into account when designing the performance component of the
LE (see below). An eventual reconciliation of the requirements informed the overall design of
the LE (Figure 2), which was conceived as being constitute from several interlinked elements:
(1) Student context – student knowledge, motivation, understanding and skills
(2) Learning context– the mode, content, scope of a dissertation and its learning objectives
(3) Supervisory context – the knowledge, motivation understanding and skills of the tutor
(4) Management context – the protocols, procedures and methods for dissertations
To emphasise on the synergistic existence of these components, a fifth element was
developed for reflecting the need for integration and acting as an essential nexus of
information and communication.
The sixth and final element recognises that performance management should not be linked to
discrete elements, but be able to monitor and report upon activities across the total set of
elements.
The figure above shows the elements of the design and distinguishes two taxonomies: one,
based on the locus of control some components must fit into a schema laid down at an
institutional level (2, 4, 5, 6) or at a programme management level (1,3,5); the other reflecting
the type of intervention, i.e. direct, involving elements 3, 4, 5 and 6, and indirect, with
elements 1 and 2.
Individually
controlled
University
controlled
4. Management
Context
2. Learning
Context
1. Student
Context
3. Supervisory
Context
6. Performance
Management
5. Integration Services
Figure 2: General Design of the Learning Environment
Direct
intervention
Indirect
intervention

Much of what was needed for the LE existed in an operational form but the different elements
were not integrated. In terms of management systems this problem was relatively easy to
address, however, the element missing from this (and for matter other educational
programmes) were the ways and means of monitoring the performance of the LE. Adopting
the mantra, ‘what gets measured gets managed’, it became clear that the Performance
component (6, Figure 2) was critical to success of an improved environment because these
indicators and measures act as a bridge between the organisational conceptualisation of the
LE and the IT and other types of system that practically implement the strategic intention
3 Developing the Performance Management Framework
In the main the plan for development of the LE performance management followed the steps
suggested by Niven (2003). A few variations were necessary to reflect the particular needs of
education and these are noted in course of the discussion below.
Step 1 - Select the Performance Framework
The need was for a performance assessment that would help to accommodate the different
facets (contexts) of the LE and be transparent to all interested parties. Three perspectives
were considered: the TQM philosophy (Widrick et al, 2002; Koch, 2003); the best-practice
adoption ideas of the European Foundation for Quality Management (Calvo-mora et al, 2005)
or the strategy-focused Balanced Score Card (BSC), (Kaplan and Norton, 2001).
Of these general approaches, the BSC was chosen as the best framework because, through
the use of strategy maps (Cardoso et al, 2005) it seemed very effective in translating the
mission of the LE into a set of operational goals and actions (Dorweiler and Yakhou, 2005).
Despite its conceptual limitations (Nørreklit, 2000) the BSC was considered to be the most
easily adaptable to the specific organisational context of higher education. It allowed to be
implemented in a mode allowing the cascade of performance management targets from
higher organisational level to the lower level, that is, from the HEI, academic school and
course team (Ruben, 1999). Finally, BSC reports could be partially automated enabling key
performance indicators (KPIs) to be monitored as ‘dashboard’ controls thus providing a
snapshot of progress at any point in time.

Step 2 - Customise the Performance Framework
The starting point in any application of analytical model and frameworks is the grounding of
the tool into the specific organisational setting. The interest of key stakeholders in the LE
formed the basis for the scorecard (Atkinson et al, 1997) arrived at is discussed in the next
section. Figure 3, shows the original form of the BSC adapted to reflect the particular
contexts for the design and implementation of the dissertation LE.
The changes to the original form of the BSC are indicated in italics. The four generic BSC
perspectives were modified in ways explained below.
A) Customer Perspective -> Student Perspective
This perspective is ‘first among equals’, in the sense that it is usually the starting point for the
development of all the performance measurements. Generically, it specifies value of the
product or service from the point of view of individuals, groups or organisations that consume
and pay for the product or a service: the ‘customers.’ In the present case, the customers are
the students and, for the LE in question, these are undergraduate dissertation students. At a
strategic level within any commercial operation it is easy to see that if you don’t meet the
needs of the customer they will seek other providers. Although this scenario is not applicable
to a HE institutions with regard to the impact of financial perspective, (B), an analogous value
proposition can be formulated in terms of the benefit derived from the knowledge capital
1. Student
context
4. Management
context
4. Management
context
5. Integration
Balanced
Scorecard –
Vision and
Strategy
3. Supervisory
context
2. Learning
context
Financial Perspective (B)
To succeed financially how should we
appear to senior management
Processes Perspective(C)
To satisfy our stakeholders what
administrative processes must we excel
at?
Learning & Growth Perspective (D)
To achieve our vision, how will we
sustain our ability to change and
improve our academic quality?
Customer Perspective (A)
To achieve our vision, how should we
appear to our students?
Figure 3: The BSC for the LE: adapted from Kaplan and Norton (1996)

transferred to the learner (Liebowitz and Suen, 2000). There are direct linkages between this
perspective and the financial (B) and developmental (D) views. The second link is to the
Learning and Growth, which concerns the inventiveness, skills, knowledge and experience
directed to create and sustain this value.
(B) Financial Perspective -> Institutional Perspective
Historically, this view has been the main influence upon corporate planning for achieving the
long term goals. Financial success is relatively easy to establish in terms of Return On
Investment (ROI), cash net profitability etc. for a business but by definition will be of limited
relevance to a not-for-profit operation (Baruch and Ramalho, 2006). The generic BSC
suggested that this perspective is all about performance judged from the point of view of
owners. The LE is owned by the University and the senior management team represent this
proprietary interest. The stakeholders particularly interested in financial performance of the
presented dissertation LE were Business School managers and, ultimately, the senior
executives team of the university. This perspective is partially located in the Management
context (4) component of the LE.
(C) Processes Perspective->Administrative Perspective
The importance of processes is that it is through these organisational systems and resources
that benefits are provided to customers. These operational activities are concerned with
operational knowledge provision in the form of service to students, academics (Pariseau and
McDaniel, 1997) and the quality assurance of the products and processes affecting both
internal and external customers. The latter responsibility includes management of the
traditional and online modes of communication. The BSC identifies the quality of internal
management processes and the deployment of resources to deliver what is expected by the
customers. In the case of the LE, this is the job of unit leaders, coordinators and support staff
who are responsible for the day-to-day conduct of the programme. This directly corresponds
to the Management context (4) component of the LE.
The performance framework shows that this is affected by the intentions and consequences
of financial performance (B) but also by the emphasis placed upon the progress,
advancement of understanding and innovation (D), the fourth and final view of the BSC.
(D) Learning and Growth Perspective -> Faculty Perspective
This perspective is important for the supervisors, since it acknowledges the effect of learning
and creativity and thus acts as a window into organisational ‘know how’ enabling the creation
of customer (student) value. This perspective acts as a window into the way the knowledge is

created, refined and exploited to yield something valued by the customer. This is not just
about the current situation but also include how technological, organisational capital can be
mobilised to ensure the organisation can sustain and improve its knowledge competencies
and capabilities in the face of environmental change. This, of course, is of primary importance
for a university because it uses knowledge to produce its ‘product’ which is knowledge
(Rowley, 2000). In the setting of the LE the stakeholders interested in, and indeed
responsible for the maintenance of the knowledge base, are the Business School academics
(the faculty), in particular those involved in the supervision the research undertaken by the
students.
Step 3 – Identify Stakeholders
Ruben (1999 p.5) argues for evaluation scheme based upon the use of multiple measures
identified from discussions with groups of stakeholder that are able to:
“Contribute pertinent and useful insights and collectively these judgements yield a
comprehensive and balance cluster of measures that help to address concerns associated
with a reliance on any single perspective or measure.”
For the present work this need was met by adapting the ideas of Olve et al (2004) concerning
four generic BSC roles: scorecard owner, change agent, information agent and measurement
(KPI) owner. Table III shows that there is a senior role within the BSC that equates to
ownership of the BSC and also represents the interest of external stakeholders, such as the
University’s Senior Management and the Senate, through a surrogate of the Head of Quality
for the Business School. The role of change agent recognises the part that should be played
by academics in the active feedback of performance into learning, knowledge growth and
subject development. Information agent is the term used to label those who are responsible
for the information underpinning each KPI.

LE Stakeholders BSC View
Contributor
Primary
Evaluation Role
Id
Deputy Dean of the Business School All BSC Owner a
Dissertation Supervisors Faculty Change Agent b
Dissertation Unit Leader Faculty Information Agent c
Dissertation & Placement Officer Administrative Information Agent e
Programme Leader Customer / Admin Information Agent d
Programme Administrator Administrative Information Agent f
Placement Advisors Administrative Information Agent g
Post-Dissertation Students Customer Information Agent h
Pre-Dissertation Students Customer Information Agent i
Head of Quality Institutional KPI Set Owner j
School Finance and Resources Officer Administrative KPI Set Owner k
Head of Undergraduate Programmes Customer KPI Set Owner l
Head of Academic Group Faculty KPI Set Owner m
Table III: Stakeholders and their Roles in LE Performance
Step 4 – Identify Critical Success Factors
After the stakeholders had been identified they were invited to attend a workshop to identify
the objectives for the areas that they represented. Each attendee was invited to suggested
three – to five goals and then to prioritise these. When this had been accomplished the goals
were represented as the first dimension of the Critical Success Factors (CSF) matrix. The
second aspect, again, completed through the use of a workshop session, was to agree the
main factors (activities and information and other resources) that most influenced the
achievement or otherwise of the goals for the LE.
The final task was to take the completed matrix and use it to understand the linkages between
all the drivers for the new LE. It was found that ‘fishbone’ diagrams (Ishikawa 2007) were an
intuitively simple way of articulating the initial cause – effect relationships, which then could
elaborated in the form of a strategy map (Cardoso et al, 2005). Figure 2 illustrates the way
academic capabilities and administrative processes help to deliver the learning experience for
undergraduate researchers and then how satisfaction with the learning outcomes contributes
to a perception of the HEI. The numbers shown for each objective correspond to the objective
identifiers shown in Table IV. These views influence the subsequent academic activity
directed toward growth and learning.

Step 5 – Develop Performance Measures
The identification of the key performance and key quality indicators was the most complex
part of developing the evaluation framework for the LE. Two general considerations
underpinned the design. Firstly, all indicators should be linked together as part of a cause and
effect chain that represents the performance of the LE. The second principle is additional to
the standard process and reflects the idea mentioned earlier about the importance of context,
by adopting the suggestion made by Buytendijk and Flint (2002) that performance
management should focus on the boundaries because these organisational interfaces is
where the main failure points lie. It will be seen that intersections between the perspectives
form dyads that map to the contexts within which performance evaluation will take place. The
beliefs shared between relevant stakeholders form the contexts and are equally important to
the design of the performance management system as the perspectives themselves, because
the context affects the interpretation of results. That is, how the measurement as information,
will become knowledge that can be used for decision-making.
Research
skills
Improved
05
Involvement
with the
community
of practice
06
Higher
Pedagogical
competency
07
Subject
specialism
enhanced
08
Improved
Skills level
01
Standardise
d
Dissertation
procedures
03
Supervisors
managed
better
04
Control of
VLE
expanded
02
Good level of
dissertation
supervision
16
Consistent
delivery and
assessment
14
Dissertation
ILO are met
more
effectively
13
Self-support
for the
novice
researcher
15
Degree
results are
improved
10
Supervisors
Productivity
is raised
11
Unit is more
economical
to run
09
Enhanced
image of the
Business
School
12
Administrative:
‘We must follow
the rules!’
Student:
‘It is the mark
that matters!’
Faculty:
‘Academics must be free
to question! ’
Institutional:
‘Move up the
league tables!’
Figure 2: LE strategy map of performance objectives and evaluative contexts

This philosophy was operationalised by articulating the values of the KPI owners and
associated information providers/users as a normative statement. The idea was that for the
objectives (and KPIs) to be meaningful, in the sense of being specific, acceptable, relevant,
the measures must be congruent with the behavioural norms of these groups, because these
would guide the attention paid to the results of the assessment and consequent actions taken.
In the setting of the LE, the KPI should flag up the extent to which collaborative or co-
operative activities as a form of collective intelligence generated by the flow of knowledge
across these interfaces (Busi and Bititci, 2006). For each of the KPI sets (Table 3) specific
assumptions were made about the type of measurement that should be used and these are
discussed below:
• Administrative
The theme for this view was evaluating the level of service provided by the LE to the
customer. The criteria were derived from the work of Abduallah (2006) on service
performance. The key indicators were those that were easy to understand by the non-
academic staff supporting the delivery of the dissertation unit.
• Faculty
Of the four perspectives, this one is best matched with the generic BSC view of
organisational learning, personal growth and innovation. However, by their nature these
processes and outcomes are quite difficult to encapsulate in a measurement not least
because some of the knowledge is personal. The stance taken was that the efficacy of
the LE for the supervisors is the extent to which it enables the absorption, diffusion,
generation and the exploitation of academic knowledge (Sprenger and ten Have, 1996).
• Institutional
These measures were based upon the work on institutional performance conducted by
Tapinos et al, (2005). The choice reflected the agenda of the senior management to
realise the potential of the staff of the university so as to improve the external image and
prestige of the university. Thus LE would be a ‘success’ if it supported the goal of the
institution to increase technical and allocative efficiency whilst maintaining the
effectiveness of undergraduate research programmes as indicated by internal and
external sources.
• Student
The measurements of the students’ views of the LE here were all related to satisfaction
with their learning experience and outcomes (Tam, 2006). It was assumed that whilst the
dissertations had a high instrumental value (encapsulated in the mark awarded) this was
not all that satisfaction is predicated upon: the dissertation management process must be

perceived as being equitable and open. This perception was derived from the difference
between expectation of the dissertation unit promulgated by the LE and what was actually
experienced by the learner. What that needed measured here is quality so it is more
accurate to call these measures for the student perspective Key Quality Indicators (KQIs)
(Browne et al, 1998).
Step 6 - Specify Performance Metrics.
Practicality was the key here by identifying simplest way of dimensioning the measurement.
The process was to look at each of the objectives shown in the strategy map and then a list of
candidate metrics were drawn up based upon existing sources of information or ones that
BSC Perspective: Administrative
Id KPI Objectives
= Service
Metric Interest
(Table 2
Source
01 Assurance Skills level of dissertation support staff c,k Opinion
02 Availability Range of VLE facilities c,k Nominal
03 Reliability Consistency of the assessment processes c,j Statistical
04 Responsiveness Turnaround time of requests for help and guidance c,k Statistical
BSC Perspective: Faculty
Id KPI Objectives
= Knowledge
Metric Interest
(Table 2)
Source
05 Absorption Number of academic enrolling on RM courses j,m Nominal
06 Diffusion Extent of communication of best research practice b,c Opinion
07 Exploitation Extent of pedagogical ideas implemented c,l Opinion
08 Generation Number of spin-off papers published c,m Nominal
BSC Perspective: Institutional
Id KPI Objectives
= Realisation
Metric Interest
(Table 2)
Source
09 Contribution Ratio of administrative cost to total unit cost a,l Financial
10 Effectiveness Average and spread of mark for the unit c,l Statistical
11 Productivity Ratio of dissertations to supervisors a,l Statistical
12 Credibility External Examiner Rating of student research skills c,l Opinion
BSC Perspective: Student
Id KPI Objectives
= Satisfaction
Metric Interest
(Table 2)
Source
13 Outcomes Rating of the learning gain from the unit c,j Opinion
14 Preparation Effectiveness of the teaching of research methods b,c Opinion
15 System Usefulness of the dissertation management system c,k Opinion
16 Supervision Effectiveness of the supervision b,c Opinion
Table IV: KPIs for LE Performance Evaluation

could be easily adapted for monitoring performance against targets. Some of these were
agreed by the unit delivery team and others were formally established by the management.
The metrics were finalised by posing and answering several questions:
(i) How important is the metric for evaluating success of the LE?
The answer revolved around the idea of whether an indicator was more ‘key’ than the
others. The conclusion was that the satisfaction ratings within student perspective
certainly met this criterion and this should be reflected in the aggregation of the points
score (see (iv) below).
(ii) Does the target time action to the LE?
Some of the KPI/KQIs reports would be required each academic term (i.e. three per year),
and the remainder linked to an annual planning cycle.
(iii) What is the referent for the target?
This question was about establishing whether a current snapshot useful or are trends in
the direction of the results were important. This was a difficult issue to resolve before the
LE was completely implemented because for some of the measures no data set existed
so the default position was single observation.
(iv) For this target can the results be aggregated with others?
This decision was about differentiating between measures by weighting each one for its
relative importance. The student satisfaction ratings, for example, could be double
weighted. Each metric would then contribute its points reflecting the degree of attainment
so that scores could aggregated per view and across whole score card to give an
overview of the LE performance. At the time of writing a decision had not been made
about this pending the validation of KPI/KQIs by the owners and users through internal
position papers and further more informal discussions.
Step 7 - Implement Measurements of LE Performance
The design of the measurement presentation ‘layer’ of the performance management regime
will be crucial for the acceptance of the measurements not only for the effect on clarity of the
results. Congruent with the earlier point about the critical nature of organisational interfaces, it
is vital to provide ways and means to enable a common understanding to be gained and to
take collaborative actions in response to the results of evaluating LE operations. Discussions
have identified the following IT enabled functions:

a. Integration of LE with general information infrastructure including university-wide
content management and future multi-media package ‘Blackboard.’ Links to personal
web pages for supervisors and those students with a common interest in a particular
subject area will also be essential.
b. Dissertation event calendars, FAQs, examples, contacts, issues lists,
announcements.
c. Automated production of a balanced scorecard and appropriate ‘dashboard’
summaries.
d. Ability to provide a common view of all the KPIs and appropriate information ‘drill
down.’
e. A feedback messaging facility enabling communication of commentaries and
observations about the performance measurement issue and problem
At the time of writing the thought, the intention for the core performance management
component (c) will at least initially be developed as an Microsoft Excel-based application that
will be driven by downloading data from the central LE. The advantage of doing this is that the
application should have a high level of familiarity, accessibility, and functionality. This should
create a high degree of stakeholder buy-in, because additional training needs will be
minimised. There are no extra software platforms have to be purchased so the Total Cost of
Ownership of the product will be relatively modest. Excel also has excellent links to other
Microsoft Office products (e.g. MS Word) and downloads/upload capability to central server-
based systems.
4 Conclusions and Reflections.
The BSC has proved its worth as a guide for managing improvements to a LE. Early results
evidenced by the expressions of satisfaction from students and supervisors; a higher degree
of consistency in assessment practice and reduction in timescale and effort of administrative
and support staff. However, the BSC has some limitations in the context of this project.
Firstly, because it was developed for business shareholders, the financial element fails to
translate well in not-for- profit organisations, such as Higher Education Institutions. That is,
where the ‘shareholders’ are the wider community, and the HEI management represents
external ownership. In the same vein, operationalisation of business strategy is not as easy to
do as for a commercial organisation where corporate and departmental profit or turnover
feature. In HEI the equivalent measurements are missing and what may be put in their place
even, using activity-based costing model, is readily contested by the different parties.
Conversely, the BSC underplays the value of human interactions, morale, motivations and
relationships within a HEI. In this sense, the BSC was an ineffective pattern to follow for LE

design. If it were possible to repeat this, it would best to shift the focus from financial capital to
something that complements the Learning and Growth perspective by fully recognising social
capital. Strategy maps (Symons et al. 2005) could be used to strengthen this aspect and to
allow the BSC to capture the dynamics of the learning environment and its alignment with
business strategy.
The current project is driven by both the need provide the best for the learners and for
educators and will help to nurture a common sense of identity between the dissertation tutors
and the tutees. It is recognised that initially, only some, perhaps just a few of the more able and
intellectually mature students will be able to actively to participate in the social network
generated by involvement with research dissertations, the so called ‘communities of practice’
(Cross et al 2001). Not only would knowledge be more visible and accessible than at present
but the development of dissertations would be driven by the collectivist views and will of the
academic participants and not so much by managerial dictate. Furthermore, new supervisors
should be able get up to speed quickly and for all academics the environment should help with
developing their skills, particularly in the critical area of assessment. Co-operation and
collaboration enabled through the LE, leading to better acquisition and utilisation knowledge of
research methods and management is supportive of the growth within the University of cross-
disciplinary sub-groups with an interest in research methods, the psychology of supervision or
assessment strategies. In such an environment, the dissertation management team will be able
to act more as knowledge brokers and facilitators, to those most directly involved in the LE.
This should facilitate the development of supervisory skills; further developing the alignment of
supervisor and student interests and resulting in more joint publications. The LE will also help to
retain organisational memory of good ideas and help to avoid costly mistakes from knowledge
loss caused by changes to personnel or lack of continuity when moving from one academic
year to the next. In all this, performance evaluation will play a vital part because it acts as vital
feedback mechanism for continued development of an LE that can deliver real benefits to the
organisation.
Some significant challenges lie ahead with regard to the operational deployment of LE
performance management. The evidence from research on the implementation of
performance management in commercial organisations (Nudurupat and Bititci, 2005)
suggests that, increased visibility and transparency of knowledge management activities are
likely to be subject to a form of ‘Uncertainty Principle.’ The act of measuring will affect the
agendas, attitudes and behaviours of academic and administrative staff and as a form of
structural modality eventually alter influence the institution (Jones and Hughes, 2001;
Koskinen, 2003). Furthermore, it is likely that the implementation of the LE and its ongoing
assessment will encounter resistance due to cultural discordance between the traditional

academic tutors and the e-oriented customers. In what way will happen, by whom and when,
could form the subject of a separate paper.
The longer-term significance of the performance system should be apparent in helping to
improve the dissertation management, by linking back outcomes directly to actions. From an
organisational point of view a significant payoff to a useful and usable LE is encouragement of
a common sense of identity between the dissertation tutors and the tutees. Not only would
knowledge be more visible and accessible than at present, but the development of dissertations
would be driven by the collective views and will of the participants and not so much by
management dictate. New supervisors could get up to speed quickly and for all academics the
environment should help with developing their competencies, particularly in the fraught area of
assessment. Co-operation and collaboration enabled through the LE should lead to better
acquisition and utilisation knowledge of research methods and dissertation management. The
dissertation management team (co-ordinator and officer) would then act more as knowledge
brokers and facilitators, to the benefit of stakeholders most directly involved in the LE. This
change should further facilitate the development of supervisory skills and may strengthen the
alignment of supervisor and student interests, which could result in joint publications. It could
also retain organisational memory of good ideas and help to avoid mistakes from any losses,
e.g. those due to changes in personnel or moving from one academic cycle to another. In this
way a truly complete learning environment for dissertations would evolve and flourish.

References
Abduallah, F., 2006, “Measuring Service Quality in Higher Education: HEDPERF versus
SERVPERF”, Marketing Intelligence and Planning, Vol. 24, No. 1, pp. 31-47.
Alavai, M., .Yoo, Y. and Vogel, D.R., 1997, “Using Information Technology to Add Value to
Management Education”, Academy of Management Journal, Vol. 40, No. 6, pp.1310-1333
Aldridge, S. and Rowley, J.,1998, “Measuring Customer Satisfaction in Higher Education”,
Quality Assurance in Education, Vol. 6, No. 4, pp.197-204.
Armstrong, M. J., 2003,“Students as Clients: a Professional Service Model for Business
Education”, Academy of Management Learning & Education, Vol. 2, No. 4, pp. 371 – 383.
Assessment & Education in Higher Education, Vol. 29, No.3, pp. 335 – 355.
Athiyaman, A, 1997, “Linking Student Satisfaction and Service Quality Perceptions: the Case
of University Education”, European Journal of Marketing, Vol. 31, No. 7, pp. 528-540.
Atkinson, A., Waterhouse, J.A. and Wells, R.B. 1997, "A Stakeholder Approach to Strategic
Performance Measurement", Sloan Management Review, Vol. 38., No.3, pp.25-37.
Baruch,Y. and Ramalho, N., 2006, “Communalities and Distinctions in the Measurement of
Organizational Performance and Effectiveness Across For-Profit and Non-profit
Sectors”, Nonprofit and Voluntary Sector Quarterly, Vol. 35., No. 1, pp. 39-65.
Becher, J.D., 2005, “Bridging the Gap between Strategy and Execution”, Business
Performance Management, March, pp 11-17.
Benson, A. and Blackman, D.,2003, “Can Research Methods Ever Be Interesting?”, Active
Learning in Higher Education, Vol. 4, No. 1, pp.39 - 55.
Booth, C. and Harrington, J., 2003, “Research Methods and Undergraduate Business
Research: an Investigation”, International Journal of Management Education, Vol.3,
No.3, pp19 -31.
Browne, B., Kaldenberg, D., Borowne, W. and Brown, D., 1998, “Students as Customers:
Factors Affecting Satisfaction and Assessment of Institutional Quality”, Journal of
Marketing for Higher Education, Vol. 8, No.3, pp. 1-14.
Busi, M. and Bititci, U.S., 2006, “Collaborative Performance Management: Present Gaps and
Future Research”, International Journal of Productivity and Performance
Management, Vol. 55, No.1, pp. 7-25.
Buytendijk, F. and Flint, D. 2002, “How BAM Can Turn a Business into a Real-Time
Enterprise, Gartner Research Notes, March.
Calvo-mora, A., Leal, A. and Roldán ,J.L., 2005, “Relationships between the EFQM Model
Criteria: a study in Spanish Universities”, Total Quality Management and Business
Excellence, Vol. 16, No.6, pp. 741-770.
Cardoso, E., Trigueiros, M.J. and Narciso, P., 2005, “A Balanced Scorecard Approach for
Strategy and Quality Universities” ; in the Proceedings of the 11th
International
Conference of European University Information Systems, June, Manchester.
Clauston,T.J. and Whitcombe, S.W., 2005, “An Emerging Person-Centred Model for Problem-
Based Learning”, Journal of Further and Higher Education, Vol.29, No.3, pp. 265 - 77.
Clayton,J., 2005, “Ask the Expert”, CIO, 23rd
May
Clerehan, R., Turnbull, J., Moore, T., Brown, A and Tuovinen, J., 2003, Educational Media
International, Vol 40.,No.1/2, pp. 17 -32.
Cross, R., Parker, A., Prusak and Borgatti, S.P., 2001, Organizational Dynamics, Vol. 30,
No.2, pp.100-115.
Doring, A., 2002, “Challenges to the Academic Role of Change Agent”, Journal of Further and
Higher Education, Vol. 26, No. 2, pp. 139-148.
Dorweiler, V.P. and Yakhou, M., 2005, “Scorecard for Academic Performance on the
Campus”, Managerial Auditing Journal, Vol. 20, No.2, pp. 138-144.
Edwards, D.F. and Thatcher, J., 2004, “A Student-centred Tutor—led Approach to Teaching
Research Methods”, Journal of Further and Higher Education, Vol. 28, No. 2, pp.195 – 206.
Gordan, P.J., 2003, “Advising to Avoid or to Cope with Dissertation Hang-Ups”, Academy of
Management Learning & Education, Vol. 2, No. 2, pp.181 – 195.
Hand, L. and Clewes, D., 2000, “Marking the Difference: an Investigation of the Criteria used
for Assessing Undergraduate Dissertations in a Business School”, Assessment &
Evaluation in Higher Education, Vol. 25, No. 1, pp.5 – 21.

Ishikawa, 2007, Ishikawa Diagra [online] Available at
http://mot.vuse.vanderbilt.edu/mt322/Ishikawa.htm
Jones, S. and Hughes, J, 2001, “Understanding IS Evaluation as a Complex Social Process:
a Case Study of a UK Local Authority,” European Journal of Information Systems,
Vol.10, No.4, pp.189-203.
Kaplan, R.S. and Norton, D.P., 1996, The Balanced Scorecard, Boston, MA, Harvard Business
Press.
Kaplan, R.S. and Norton, D.P., 2001, “Transforming the Balanced Scorecard: from
Performance Measurement to Strategic Management: Part 1”, Accounting Horizons,
V15., No.1, pp. 87-104.
Kim Hua, K. Tan, K., Platts, K. and Noble, J., 2004, “Building performance through in-process
measurement: Toward an ‘indicative’ scorecard for business excellence”,
International Journal of Productivity and Performance Management, Vol. 53, No.3,
pp.233-244.
Koch, J.V., 2003, “TQM; Why is its Impact in Higher Education so Small?”,TQM Magazine,
Vol.15, No.5, pp. 325-333.
Koskinen, K., 2003, “Evaluation of Tacit Knowledge Utilization in Work Units”, Journal of
Knowledge Management, Vol 7.,No.,5, pp. 67-81.
Liebowitz, J. and Suen, C.Y., 2000, “ Developing Knowledge Management Metrics for
Measuring Intellectual Capital”, Journal of Intellectual Capital, Vol.1, No.1 pp. 54 -67.
Mackinnon, J., 2004, “Academic Supervision: Seeking Metaphors and Models of Quality”,
Journal of Further and Higher Education, Vol.28, No.4, pp.395 - 405.
Milton,J. and Lyons, J., 2003, “Evaluate to Improve Learning: Reflecting on the Role of
Teaching and Learning Models”, Higher Education Research & Development, Vol.22,
No.3, pp.297 – 312.
Niven, P., 2003, Balanced ScoreCard: Step-by-Step for Government and Non-Profit
Agencies, John Wiley and Sons, New York, N.Y.
Nørreklit, H. 2000, "The Balance on The Balanced Scorecard – A Critical Analysis of some of
its Assumptions", Management Accounting Research, Vol.11, No.1, pp. 65-88.
Nudurupat, S.S. and Bititci, U.S.,2005, “Implementation and Impact of IT-Supported
Performance Measurement Systems, Production Planning & Control, Vol.16, No.2,
pp. 152-162.
Oldfield, B. and Baron, S., 2000, “Student Perceptions of Service Quality in a UK Business
and Management Faculty”, Quality Assurance in Education, Vol.8, No.2, pp. 85-95.
Olve, N-G., Petri, C-J. and Roy, S., 2004, “Twelve Years Later: Understanding and Realizing
the Value of Balanced ScoreCards”, Ivey Business Journal, University of Western
Ontario, May/June, pp.1-7.
Pariseau, S.E. and McDaniel, J.R. 1997, ``Assessing service quality in schools of business'',
International Journal of Quality & Reliability Management, Vol.14 No.3, pp.204-218.
Ramsden, P., 1992, Learning to Teach in Higher Education, London, Routledge.
Rees, P. and Gardner, H 2003, “Best Value, Partnerships and Relationship Marketing in Local
Government”, International Journal of Non Profit and Voluntary Sector Marketing, v.8,
Iss.2, pp. 143-152.
Rowley, J., 2000, “Is Higher Education Ready For Knowledge Management?”,International
Journal of Educational Management Vol.14, No.7, pp. 325-333
Rowley, J., 2004,“What is the Future for Undergraduate Dissertations?”, Education &
Training, Vol. 46, No. 4/5, pp.176-181.
Ruben, B.D., 1999, “Toward a Balanced Scorecard for Higher Education: Rethinking the
College and University Excellence Indicators Framework”, QCI Center for
Organizational Excellence, Rutgers University at www.qci.rutgers.edu.
Sanger, M, 1998, Supporting the Balanced Scorecard, Work Study, Vol.47, No.6,pp.197-200.
Sprenger, C. and Have, S.ten, 1996,”The Role of Knowledge Management in the Learning
Organisation, Holland Management Review, September-October, pp. 73 -89.
Symons, C., Orlov, L., Bright, S. and Brown, K., 2005, IT Strategy Maps: A Tool for Strategic
Alignment, Forrester Research, November 21.
Tam, M., 2006, “Assessing Quality Experience and Learning Outcomes”, Quality Assurance
in Education,Vol. 14,No.1, pp. 75-87.

Tapinos,E., Dyson, R.G and Meadows, M., 2005, “The Impact of the Performance
Measurement Systems in setting the ‘Direction’ in the University of Warwick”,
Production Planning and Control, Vol. 16, No.2, pp. 189-198.
Taylor, J., 2001, “Improving Performance Indicators in Higher Education: the Academic’s
Perspective”, Journal of Further & Higher Education, Vol. 25, No.3, pp.379-393.
Todd, M., Bannister, P. and Clegg, S., 2004, “Independent Inquiry and the Undergraduate
Dissertation: Perceptions and Experiences of Final-Year Social Science Students”,
Todd, M., Smith, K. and Bannister, P., 2006, “Supervising a Social Science Undergraduate
Dissertation: Staff Experiences and Perceptions”, Teaching in Higher Education,
Truyen, F. and Van Rentergem,L., 2005, “Preparing the University Information Architecture for
Netcentric E-learning and Research: a case-study:” in the Proceedings of the 4th
European Conference on e-Learning, November, Amsterdam.Vol.11, No.2, pp.161-173.
Webster, F, Pepper, D. and Jenkins, A., 2000, “Assessing the Undergraduate Dissertation”,
Assessment & Evaluation in Higher Education, Vol. 25, pp. 71 -80.
Widrick, S.M., Erhan, M. and Delven, G., 2002,“Measuring the Dimensions of Quality in
Higher Education”, Total Quality Management, Vol.13, No.1, pp.121-131.
Acronyms
BPM Business Performance Management
BSC Balanced Score Card
CSF Critical Success Factors
FAQ Frequently Asked Questions
HE Higher Education
HEI Higher Education institutions
IT Information Technology
KPI Key Performance Indicators
KQI Key Quality Indicators
LE Learning environment
RM Research Methods
ROI Return On Investment
TQM Total Quality Management
VLE Virtual Learning Environment

Appendix A - Problems with the Dissertation Learning Environment before the intervention
ƒ Tracking of supervisors’ performance
ƒ Communication within the team
ƒ Collection of data is not agreed with the supervisor
ƒ Limited use of the Virtual Learning Environment
ƒ Turnaround of assessment
ƒ Sporadic conformance to requirements
ƒ Tracking student performance
ƒ Inconsistent marking
ƒ Inconsistent feedback to students
ƒ Inconsistent level of support amongst supervisors
ƒ Documentation not standardised across courses
ƒ Allocation of dissertations supervision happens last and is sometimes given to people with no/little research
experience
ƒ Limited transfer of best practice
ƒ Inadequate documentation of the processes and procedures in place
ƒ Supervisors with limited knowledge of research methods theory
ƒ Workload allowance does not reflect the investment of time per student
ƒ Working with students with Additional Learning Needs
ƒ Limited knowledge of the support provided to students
ƒ Limited knowledge of what information has been disseminated to students.
ƒ Wide variety of topics with limited expertise in-house
ƒ Expectations of supervisor’s availability and turnaround of feedback unrealistic.
ƒ Difficulties with estimating the scope of the project and managing the work
ƒ Cases of academic offence
ƒ Inadequate knowledge of Research Methods
ƒ Limited knowledge of the research process
ƒ Choice of topics digress

Applying Performance Management Principles To A Learning Environment For Undergraduate Dissertations A Case Study

Recommended

Recommended

More Related Content

Similar to Applying Performance Management Principles To A Learning Environment For Undergraduate Dissertations A Case Study

Similar to Applying Performance Management Principles To A Learning Environment For Undergraduate Dissertations A Case Study (20)

More from Laurie Smith

More from Laurie Smith (20)

Recently uploaded

Recently uploaded (20)

Applying Performance Management Principles To A Learning Environment For Undergraduate Dissertations A Case Study