Integration of technical development within complex project environment

i
Integration of technical development within complex
project environments
Project
Department of Computer Science at the University of York
Author: Nick Brook
Project Supervisor: Katrina Attwood
Abstract
The integration of design and safety functions within complex technical system development and project
methodologies is still undeveloped. Deficits in management are primarily manifested through large numbers of
major unanticipated changes. This variously results in overrunning costs and schedule and compromised fulfilment
of important project requirements. These undesirable consequences are greatly magnified by complexity.
This project will attempt to address this deficit. The high-level objectives are to develop a framework to facilitate
better planning, monitoring and control of technical activities. This will be achieved through the identification,
development and integration of suitable existing concepts and techniques into a complexity management
framework. This should apply to all complex projects and particularly those relating to safety critical engineering
projects. Primarily, the project builds upon the research which has been previously undertaken by this author in
project failings, complexity and existing tools and techniques.
The framework will be evaluated through questionnaire survey of several of its component parts, a review against
the recommendations from literature and through case study application of the tools and techniques. It is the goal
that all or parts of this framework can be put to practical application within industry.
Statement of Ethics
This dissertation, including literature research, questionnaire survey and conclusions, has carefully considered and
adhered to the three principles of ethics specified by the University of York. These are - to do no harm, to ensure
informed consent from human participants and to uphold sound principles of data confidentiality:
Do No Harm
No physical system or entity has been developed which could cause harm of any kind. The work undertaken
pertains to literature research, the gathering of non-sensitive and non-confidential data via a questionnaire
survey, and the development of research conclusions.
Informed Consent
Information which is freely available in the public domain has been appropriately referenced within this
dissertation. Some information was acquired by direct requests to, and discussions with, the authors; at all times
the authors have been made aware of the purpose of the request and the nature of the critical evaluation.
Questionnaire respondents freely participated and were made aware of the purpose of the survey beforehand.
Confidentiality of Data
No sensitive or confidential data has been acquired, used or released during the course of this dissertation.
Number of words is 39,473 as counted by the MS Word word count. This includes all of the body of the report, but
excludes the appendices which are included in the project submission for completeness and interest.

Nick Brook MSc Safety-Critical Systems Engineering 9th
January 2017
ii
Left intentionally blank

January 2017
iii
Table of Contents
Table of Contents........................................................................................................................iii
Acknowledgements.....................................................................................................................iv
1. Introduction .........................................................................................................................1
1.1 Overview............................................................................................................................................ 1
1.2 Objectives.......................................................................................................................................... 1
1.3 An understanding of complexity ....................................................................................................... 3
1.4 The importance of considering the organisation .............................................................................. 4
1.5 Existing management frameworks.................................................................................................... 5
1.6 Devising the Complexity Management Framework.......................................................................... 8
2. Identifying and quantifying project complexity......................................................................8
2.1. The complexity assessment matrix ................................................................................................... 8
2.1.1. Complexity themes.................................................................................................................... 9
2.1.2. Complexity Criteria.................................................................................................................. 10
2.1.3. Complexity Matrix ................................................................................................................... 11
2.2. How, where and when to assess..................................................................................................... 14
2.3. Outputs from complexity assessment............................................................................................. 15
2.4. Complexity profile and the interaction of complexity criteria........................................................ 15
3. Critical Project Success factors and their application in system development .......................17
3.1. Selection of success factors from literature.................................................................................... 17
3.2. Verification of success factors through questionnaire.................................................................... 18
3.2.2. Full dataset .............................................................................................................................. 19
3.2.3. By age ...................................................................................................................................... 22
3.2.4. By role...................................................................................................................................... 22
3.2.5. By industry............................................................................................................................... 23
3.2.6. Conclusion ............................................................................................................................... 24
4. System development planning techniques...........................................................................24
4.1. Overview.......................................................................................................................................... 24
4.2. Design Structure Matrix................................................................................................................... 25
4.2.1. Introduction............................................................................................................................. 25
4.2.2. Design Structure Matrix principles.......................................................................................... 26
4.2.3. Creating and applying the organisational architecture DSM .................................................. 27
4.2.4. Creating and applying process architecture DSM ................................................................... 27
4.2.5. Application of the process DSM within a complexity management framework..................... 30
4.2.6. Process-organisational MDMs and their application .............................................................. 33
5. System Performance Measures ...........................................................................................34
5.1. Introduction..................................................................................................................................... 34
5.2. Desirable properties of Performance Measures ............................................................................. 35
5.3. Selection of System Performance Measures from literature.......................................................... 38
5.3.1. Requirements .......................................................................................................................... 38
5.3.1.1. Satisfaction of Stakeholder and System Requirements .......................................................... 38
5.3.1.2. Requirements attributes ......................................................................................................... 39
5.3.2. Development health................................................................................................................ 40
5.3.3. Process maturity...................................................................................................................... 40
5.3.4. System maturity....................................................................................................................... 41
5.3.5. Organisation, process and schedule complexity ..................................................................... 41
5.4. The verification of performance measures through questionnaire................................................ 42
5.4.1. Performance measures by age, role and industry................................................................... 43

January 2017
iv
5.4.2. Conclusion ............................................................................................................................... 43
5.5. The collective use of performance measures ................................................................................ 43
6. Managing system development risks...................................................................................44
6.1. Introduction..................................................................................................................................... 44
6.2. Important concepts ......................................................................................................................... 45
7. Complexity orientated development framework .................................................................46
7.1. Integrating the sub-processes together .......................................................................................... 46
7.2. Analysis of framework against criteria within existing literature.................................................... 49
8. Case study application of framework...................................................................................50
8.1. Purpose............................................................................................................................................ 50
8.2. Case study 1 – Boeing Dreamliner................................................................................................... 50
8.2.1. Background.............................................................................................................................. 50
8.2.2. Work Breakdown Structure..................................................................................................... 51
8.2.3. Complexity assessment ........................................................................................................... 53
8.2.4. Critical Success Factors............................................................................................................ 56
8.2.5. Planning technique.................................................................................................................. 58
8.2.6. Performance measurement .................................................................................................... 58
8.2.7. Risk management .................................................................................................................... 59
8.2.8. Comparing framework findings with project outcomes.......................................................... 59
9. Results and Evaluation........................................................................................................60
10. Conclusion..........................................................................................................................64
References…………………………………………………………………………………………………………………………………..66
Appendix A Glossary of terms and acronyms
Appendix B Existing management frameworks
Appendix C Critical Success Factors
Appendix D Response summary
Appendix E Complexity questionnaire
Appendix F Questionnaire results ranked by influence
Appendix G Dataset sample sizes
Appendix H Questionnaire results filtered by age
Appendix I Questionnaire results filtered by role
Appendix J Questionnaire results filtered by industry
Appendix K Design Structure Matrix
Appendix L System Performance Measures
Appendix M Risk register template
Appendix N Risk attributes as metadata
Appendix O Criticality Assessments for Boeing Dreamliner case study
Appendix P Timeline to Boeing Dreamliner
Appendix Q Complexity management case study using OL3 and AREVA’s EPR
Appendix R Full questionnaire results per respondent
Acknowledgements
I would like to thank Lana and Anna for coping with my extended period of further education and putting up
with my many grumpy moods. I also have the utmost respect for the University of York and the teaching and
administrative staff who have made my studies so enjoyable and rewarding.
Finally, my Mum and Dad have been extremely supportive and I could not have done this without them.
Continuous effort - not strength or intelligence - is the key to unlocking our potential. - Winston Churchill.

January 2017
Page 1 of 70
1. Introduction
1.1 Overview
The characteristics of complex systems include a difficulty in determining outcomes from the inputs [1] and
there being present a ‘degree of disorder’ [2]. The relationships and dependencies between processes,
organisations and other external systems describe perfectly that of any complex system. What is more, the
general trend is for development complexity to increase over time. This is while simultaneously catering for
advances in technology and the downward pressures on development timescales and cost. The result is
greater prevalence of the wicked problem, that is a technical system development that is ‘highly resistant to
resolution’ [3]. This can be thought of as Technical Development Complexity.
This dissertation builds on the research undertaken within the previous literature survey [4]. This work
described the high failure rate amongst projects, a summary of the main reasons for this and areas of further
investigation. Also a variety of definitions were given for complexity and compelling reasons that this should
be the focus of attention when planning for the development of a system. The dissertation develops the
literature survey’s conclusions, describing a framework for the treatment of complex system development
within a wider project environment.
As the network of individual activities and their interactions increase it is reasonable to expect that the
mechanisms that are used to achieve a successful outcome will be greater in terms of their magnitude and
resources to undertake them. Therefore, complexity inevitably influences the effort required to plan, monitor
and control development activities. These mechanisms and conditions can be specified within development
processes but it would appear reasonable that these also need to be tailored to suit the particular
development characteristics and especially those relating to its complexity. These are known as Critical
Success Factors (CSF) and describe ‘essential areas of activity that must be performed well if you are to
achieve the mission, objectives or project’ [5]. However, it should be noted that outside describing their form
and importance, literature provide no satisfactory methods or process for determining suitable CSFs.
1.2 Objectives
It will be proposed that complexity comprises a number of aspects and that, for the purpose of this
dissertation, these can be comprised of themes and criteria. These aspects will not be homogeneous across
all the development activities nor the development lifecycle. This variation across a project can be seen as a
complexity profile that will evolve throughout the development lifecycle. Outwardly similar developments,
even within the same organisation, may not exhibit the same complexity characteristics. This can often be
attributed to the presence (or lack) of constraints outside the development team’s control and the result of
influential management decisions. Consideration of complexity will have a number of goals:
 Identify the most influential Critical Success Factors (CSF) to put in place environmental conditions
required for a successful outcome;
 Identify methods to put the responsibility for interventions where it is best placed through early
identification the establishment of environmental factors is often beyond the direct influence of the
development team or its managers [6][7];
 Recognise how complexity evolves throughout the development and how some aspects increase as
others diminish. As such the methods to plan, monitor and control development activities will need
to evolve accordingly throughout the development lifecycle;
 Describe how interventions can influence complexity and displace its effects elsewhere;
 Recognise and anticipate development risks identified through the consideration of complexity early
in the development lifecycle and during detailed planning activities;
 Develop a framework for assigning interventions to defined risks more closely, at the appropriate
level within the Work Breakdown Structure (WBS). The identification of interventions will be initiated
by the recognition of CSFs and risk planning;

January 2017
Page 2 of 70
 Propose methods to guide where the most effort should be applied in planning, monitoring and
controlling activities through the identification of areas of maximum complexity. This is where the
greatest number of residual risks is likely to be. This is most important where there exists a constraint
of available resource which inevitably means effort must be prioritised. This is a factor of varying
significance in all but the most exceptional of technical developments;
 Ensure the framework is scalable with regards to the size of the development and to the level within
the WBS Structure to which it can be applied. This is closely related to the preceding item;
 Provide a framework to supplement planning techniques to better capture aspects of complexity,
such as coupling behaviours between activities. This will also allow analysis to identify individual
activities or clusters of planned activities that have the potential to influence the overall
development outcome disproportionately and as such deserve enhanced management effort;
 Guide the selection and implementation of development status measures. These should be
determined on the basis of the impact of particular areas of technical development and the key
activities within these areas. These should be selected primarily for their use in controlling the
development and initiating avoiding action and not merely for purpose of reporting status.
An underlying theme of the framework will be that the activities it directly influences (planning and
monitoring), and indirectly influences (controlling), are chosen to balance risk and benefit with effort
involved. It should complement current techniques, and absolutely not conflict with them. It is the aim that
this will allow their use with a minimum of additional resource overhead. It is recognised that the framework
will not be implemented in isolation of existing project management and system engineering tools and
techniques. A secondary objective is to allow project and process complexity to be understood through
practical usage of the techniques. This can be achieved through the feedback of learning back into the process
to improve it and develop better responses for future system development.
There are often latent issues within system development that are only manifested through delays, cost
overruns or quality issues. This is problematic for several reasons. The intervention required is generally
greater the later it is left. Also as the development tempo intensifies, the effort required to change the
process and organisation will inevitably draw on the very same resources as those developing the system
itself. The disruption and additional uncertainty of making these changes will exacerbate the impact of the
original latent issue. In fact, the scope of such a change can be seen as a project in its own right [8] and is as
undesirable as it is unnecessary. Following on, there is evidently great benefit in balancing the effort between
undertaking risk management as opposed to that of issue management. This balance is often forgotten and
especially where there is insufficient substantiating data to support the mitigation of risks or to support
change. The focus instead is on constant management of issues arising from bad risk planning, commonly
known in project management as ‘firefighting’ [9], leading to no long-term sustained improvements in
performance.
A goal of the framework is to provide strong enough indicators to prompt this early management action.
Many of the principles in the management of system related risks are equally applicable to project risks. This
can be illustrated by an annotated Bow Tie diagram [10][11] as shown in Figure 1. The role of complexity
management is both to identify the threats on the left hand side that are brought about by complexity and
to determine effective methods for controlling them.
Selecting CSFs should be followed by an understanding of residual risks and ways of providing an early
warning of their realisation. A semi-formal method for identification of CSFs within a framework will
encourage their wider use and raises the possibility of their use at development and project review points,
such as design reviews and stage gates. An objective should be the early identification and best possible
assignment of responsibility for the of the mitigation of development risks, not only across the breadth of
the project but to a sufficiently senior level of management [12].

January 2017
Page 3 of 70
Heuristic techniques will be chosen and developed as a means of guiding decision making and providing
inputs into traditional project management and system engineering processes. These will anticipate areas
within system development that require the most effort and where effort will bring about the greatest
impact. This will provide multiple Plan- Do-Check-Act (PDCA) cycles, an iterative four-step management
method used to control and allow continual improvement of processes and product development [13]. Also
known as the Demming Cycle, it is equally applicable within the development process. In this application it
will be used to influence the development environment, processes and planning for complex system
development. The techniques described will use a mix of decomposition and reductionist methods, along
with a heuristic approach, to optimise resource usage required both of the framework and by any additional
development effort that results from its application.
Heuristic techniques are well suited to the problems associated with planning and executing a complex
system development and work well alongside iterative development techniques generally advocated for
complex system development. There are many variables in play within complex technical development.
While optimal solutions are achievable for a particular aspect, optimisation across all these often competing
aspects is very difficult within the current methodologies.
An important concept is that of feeding back learning, into the on-going development or subsequent
developments, to improve future applications of the framework. The determination of CSFs or risks should
include the intended or optimal outcome which can be compared against actual outcomes during the
development lifecycle for the efficacy of the CSFs and their implementation to be assessed. This allows
improvements to be made in a timely fashion for the benefit of the development. It also avoids the
unsatisfactory analysis of lessons learned, often at the very end of the project when personnel drift away and
the motivation to hold a review wanes [14].
A glossary of terms and acronyms used throughout this dissertation can be found in Appendix A.
1.3 An understanding of complexity
Complexity must be understood before it can be modelled. It has previously been described and decomposed
in a large number of ways, both within and outside the domain of engineering and many other overlapping
fields of research. Examples include the Strategic Highway Research Program [15] and Helmsman Institute
[16]. For the purpose of this framework it is important for complexity to be represented as simply as possible,
while covering all the necessary aspects as effectively as possible. This may appear counterintuitive but is
vital if the principles of the framework are to be applied in practice. This section will summarise research on
the nature of complexity and will develop it for use in a complexity management framework. A more detailed
explanation of the properties of complexity can be found within the literature survey [4].
Figure 1. Annotated Bow Tie diagram [11].

January 2017
Page 4 of 70
Definitions of complexity highlight is highly structured nature [17][18] which encompasses many interactions
[19] and a large array of possible configurations [20][21]. Other characteristics include there being no
relationship between the behaviour of the overall structure and that of its individual parts and the ‘response’
of system elements to changes to other interrelated system elements [20][21][22]. Together these aspects
make the overall structure difficult to understand [23]. It is noted that uncertainty is often omitted from the
definition of complexity though it is intertwined with several of the aforementioned characteristics [24], such
as configuration variation. This project proposes that ambiguity be considered [25] as a distinct and separate
characteristic, alongside Uncertainty. Ambiguity is a common property of system development, caused by
the absence or unavailability of reliable information, and resulting in the formation of assumptions.
Ambiguity is a dominant factor in the early stages before requirements have been agreed and also during
development phases before coupling of large numbers of activities are fully understood. Assumptions cannot
be completely validated until later in the development and only after ambiguity has been addressed.
An understanding of how the individual characteristics of complexity interact with each other will be pivotal
to the determining of CSFs. Further dimensions of complexity will be elaborated upon in Section 2.
1.4 The importance of considering the organisation
The main two components of a system
development, apart from the actual system being
developed, are the development process and the
development organisation. These can be
considered to be within the direct control of the
management team responsible for system
development. Additionally, there will be a number
of external interfaces and constraints that will be
outside direct control of the system development
management team. These may be from within the
actual project itself, such as overall project budget
or resource availability, or entirely outside of the project.
Examples of the latter may be stakeholders, regulators and
existing adjacent operating systems.
An understanding of these boundaries will be important in
assigning responsibility for interventions. Figure 2 shows the
British Computer Society complexity model with tiered levels
of external interface. This is a simplified representation and
does not show how these tiers interact but nevertheless is
useful in categorising project influences. Specifically, the
model includes the ‘macro-environment’, which
encompasses interfaces furthest from the control of the
development team such market condition, legislation and
regulatory frameworks. ‘Micro-environment’ relates to the
customer and stakeholders and how requirements are
captured and traded-off [26]. The ‘organisational
environment’ and the ‘project environment’ relate to the
system development and the resource and process
constraints imposed on it. The organisational relationship
Figure 2. The British Computer Society’s
complexity model [26].
Figure 3. Type of relationship between
engineering and project management teams
(adapted from SEBoK [27]).

January 2017
1
The ‘Iron Triangle’ describes The three related constraints of scope, budget, and schedule [32].
Page 5 of 70
between system engineering and the project environment is also an important consideration and something
which this model omits. Figure 3 shows three distinct project organisational types from the viewpoint of the
technical development and is adapted from the Systems Engineering Book of Knowledge (SEBoK) [27].
These organisational types strongly influence what is and what is not within the direct control of the system
engineering team. The definition of the inputs and outputs between project and system development
organisations will allow a better understanding of where the responsibility for putting in place and
implementing CSFs and risk mitigations resides. These interfaces should be defined within such documents
as the Systems Engineering Management Plan [28] and the Project Management Plan [29].
In the first diagram organisational behaviour is typified by transactional type relationships. Engineering may
be contracted out or undertaken by a different organisation, possibly located elsewhere. There may be more
than one engineering organisation, with complexity increasing according to the number of interfaces. The
project management organisation is responsible for managing the outputs only and there is a strong
emphasis on functional organisational structures [30]. In the second diagram the overlap in responsibilities
and accountabilities can also result in a form of complexity.
There is a lesser emphasis on transactional relationship and the project management team have a greater
influence over the management of engineering activities. The degree of overlap will affect this relationship.
The third diagram could be a matrix type structure. The project management team has overall control and
the engineering management team has the least scope to determine how the development will be managed.
The type of relationship will thus affect the scope and the method of managing complexity. In the first
example system engineering is entirely self-sufficient in terms of contractual complexity. In the last example
this is far less likely. Thus the treatment should be amended accordingly.
1.5 Existing management frameworks
There has been a large volume of literature on the subject of managing projects and technical development
activities based on a blend of theory and practical experience. The framework presented in this project will
take the existing management frameworks and models, many discussed in the literature survey [4], and will
attempt to reconcile inconsistencies in approach and content. This section will both consolidate the
discussion in the literature survey [4] and introduce concepts that have been found subsequently.
The predominant systems engineering model is the ‘V-model’ as shown in Figure 4. It does not specifically
address complexity but has some concepts that will be applied within the developed framework. Specifically,
its definition of the development lifecycle and requirements-centric view is useful. The satisfaction of
requirements is a convenient way of looking at complexity and the monitoring and control of requirements
should form an integral part of the way that a complex system is managed. The V-model represents a typical
idealised development lifecycle, so the phase descriptions within this model will be used to profile
complexity.
Of the system engineering and project
management methodologies that were
previously discussed in the literature survey [4]
the Strategic Highway Research Program’s
5DPM was of particular interest. This
methodology is described in some detail in both
their ‘Guide to Project Management Strategies
for Complex Projects’ [31] and also in ‘Managing
Mega-Project Complexity in Five Dimensions’
[31]. Rather than considering the usual three
project dimensions, i.e. the iron triangle1
[32], it Figure 4. The V-model [26].

January 2017
Page 6 of 69
introduces the concepts of ‘financing’ and ‘context’ as shown in Figure 5. 5DPM also considers CSFs from the
perspective of complexity. Furthermore, this is done within several feedback cycles. Together these
represent considerable synergy with many of the concepts that the author previously felt were worthy of
consideration.
Finance becomes of utmost consideration as both the overall magnitude of the development and risk
increases. This is exemplified by the difficulties experienced in securing investment for the UK’s nuclear new-
build programme [32]. For the purposes of technical development, it is an imposed constraint by virtue of
being outside the direct control of the management and will be treated as such in the model. The fifth
constraint of the 5DPM model is that of context, describing the project environment, including potential
constraints and interfaces. The process can be decomposed as
follows:
 Review project factors in each of the five areas;
 Identify and prioritise complexity factors;
 Develop 5DPM complexity map;
 Define CSFs with sub-process steps such as assemble
project team, select project arrangements and prepare
early cost model and finance plan;
 Develop project action plan to address resource issues;
 Re-evaluate complexity map on commencement of
project.
5DPM describes reviews at defined intervals with an iteration of the above steps. This would most naturally
align with a Stage Gates type governance structure [29], but it could be envisaged that interim reviews would
be undertaken on a periodic basis if stage gates were deemed too far apart for effective control. Overall the
model does not try to model complexity itself but rather attempts to model the component parts (within the
five dimensions) and manage them to better manage a complex project environment. By focussing on these
five dimensions the resulting assessments may provide a superficial view of project management orientated
aspects only. Despite this there is considerable merit to the high-level approach and synergy with the
principles of the PDCA Cycle [4].
The Helmsman Institute complexity assessment [16] is advocated by the International Centre for Complex
Project Management [34]. This again has five areas of general consideration consisting of:
 Context Complexity;
 People Complexity;
 Ambiguity;
 Technical Challenge;
 Project Management Complexity.
Within these categories there are specific factors. Those solely relating to technical development are:
 Integration Complexity;
 System Development Complexity;
 Impact on Infrastructure.
Each of the categories is in turn considered against a large number of factors relating to such concepts as
stakeholder numbers and alignment, uncertainty and abstraction. Importantly this framework introduces the
concept of ambiguity as does Pich et al [25] and McGowana et al [35]. In each ambiguity is seen as a distinctly
separate, though closely coupled, property to that of uncertainty. The consideration of complexity is of
interest within the wider domain of business in general. The concept of VUCA (volatility, uncertainty,
complexity and ambiguity) [36], as shown in Figure 6, considers many of the previously discussed themes.
Volatility can be viewed as the result of one or the product of several complexity characteristics. Traits
attributable to volatility include a general lack of understanding of the likely impacts of an event
Figure 5. Five dimensional
project management model [15].

January 2017
Page 7 of 70
along with the impacts being
‘unexpected’ and ‘unstable’.
Here volatility is described as a factor in
its own right and complexity is described
alongside the others rather than being
composed of them. Nevertheless, the
presentation of an unpredictable
business environment in terms of a
limited number of characteristics is
useful. Much of the literature references
complexity without sufficiently breaking
it down into its component parts.
Classical definitions of the components
of complexity were described detail
within the literature survey and are as
follows [4][37]:
 The ‘Kolmogorov-Chaitin
complexity’, also known as
Program-size Complexity [38];
 ‘Nonlinearity’;
 ‘Emergent Complexity’.
Essentially the first component is the number of interfaces relating somewhat to the development magnitude
but also its nature. A large technical development may contain duplicate tasks and low interdependency,
thus limiting its Program-size Complexity, while a comparably smaller technical development may conversely
have many interdependencies and highly intricate activities. The latter two components collectively can be
closely aligned with the previously described volatility within the VUCA framework. A relatively small change
results in far-reaching or a disproportionately large impact. Expanding the definition of emergence for the
benefit of its use in an assessment we may say that it has a number of properties [39]:
 Radical novelty – new and unanticipated properties emerge from each interaction;
 Coherence – though unexpected the new properties are consistent with rules and behaviours;
 Wholeness – emergence causes the new system that is greater than the sum of its parts;
 Dynamic – changes from emergence will continue to evolve as long as there is emergence;
 Downward causation — the system as a whole dictates the behaviour of its parts as well as the
system being influenced by the interaction of its parts.
This may be caused by two common dynamics, either separately or in conjunction:
 No one is in charge – thinking in terms of a technical development organisation this could be
characterised by a network-centric organisation with a decentralised structure;
 Simple rules engender complex behaviour – using the example of an organisation this may be a large
hierarchical organisation with many functions.
This section describes the decomposition and treatment of complexity from varying perspectives. The
development of the 5DPM process supports the view that simple consideration of the ‘iron triangle’ is
insufficient. It also reinforces the importance of complexity and the validity of an iterative approach.
However, its treatment of complexity is overly simplistic. The complexity framework developed in this project
will adapt the broad methodology of the 5DPM process, while proposing a more precise definition for
development complexity using a combination of prevailing theory and practical application. Specifically,
definitions of complexity outlined either in the literature survey or in this section will be re-categorised to fit
within the model, maintaining consistency.
Figure 6. The VUCA framework [36].

January 2017
2
The objectives described in the summary of the literature survey [4] will be adopted with a single change. The
development of Object Based Modelling (OBM) will be omitted. DSM will be used alone as a flexible technique that can
be used to model the development process. This of course does not preclude the use of techniques, such as OBM, in
parallel with DSM. Further information on the use of OBM can be found in the paper by Warboys and Keane [41].
Page 8 of 69
Complexity will be decomposed by applying it against the relevant themes within the technical development.
It must also be considered that complexity is in the eye of the beholder. The observer will have experience
and a perspective that will influence subjective analysis based on their ‘bounded rationality’ [40]. The
assessment criteria will ultimately require to be qualified as much as possible to guide interpretation. A more
detailed literature review of this area of research can be found in Appendix B.
1.6 Devising the Complexity Management Framework
This dissertation will define complexity in terms of technical development and combine a method of assessing
complexity with a number of techniques to allow the planning, monitoring and controlling of complex
technical developments. To summarise this dissertation will2
:
a. Identify a method of quantifying Technical Development Complexity;
b. Provide a semi-formal method for the identifications of the pre-requisite pre-planning environmental
factors, or so called critical success factors (CSFs). The outcomes from this method will be influenced
by an individual project’s characteristics, primarily in terms of its complexity;
c. Propose methods for planning technical development and modelling interdependencies using the
method of Design Structure Matrix (DSM);
d. Provide Performance Measures that relate to the technical development and, where possible, to the
assigned CSFs. It is preferable that these are totally objective but it is accepted that there will be
considerable subjectivity involved which will need to be constrained to enhance consistency by
suitable guidance. These performance measures will be both leading and lagging and provide
coverage of the development process biased towards the parts that are particularly significant for
reasons either of criticality or of risk of deviation in terms of criteria such as cost, schedule, scope
and quality. Coverage of the remaining parts of the development process will be at a lower degree
of decomposition commensurate with criticality and risk;
e. Propose methods of identifying risks and how this influences CSFs and the Performance Measures.
f. Demonstrate how all these (a. to e.) can be used together in a Complexity Management Framework.
Together items a. and b. can be thought of as pre-planning activities and together are a strategy for putting
in place an environment conducive for the project. Items a. to e. are developed in more detail throughout
this project, culminating in Section 7, which includes a framework map integrating all these items together.
2. Identifying and quantifying project complexity
2.1. The complexity assessment matrix
The complexity concepts described above will be encompassed within a Complexity Assessment Matrix. The
matrix will form the most important part of the Complexity Assessment which in turn forms a part of the
overall framework. The outputs from the assessment are as follows:
1. Undertake assessment of system development at a high-level within the WBS;
2. Identify of ‘hotspots’ both within the WBS and within the individual themes and direct additional
assessments where they are required. It should be noted that focussed assessments will not be
possible in very early development stages due to the low level of definition and decomposition of
the WBS;
3. Identify development risks and propose CSFs to counter significant levels of complexity. Record
additional information such as:
a. Rationale;
b. Residual risks that remain once CSFs have been put in place;
c. Proposed outcomes that are to be brought about by the CSFs.

January 2017
Page 9 of 69
4. Reassess the project complexity profile with the CSFs in mind to ensure that no new significant
unmitigated hotspots have been created. Revise the outputs of step 3 as appropriate;
5. Identify risks for inclusion within the risk register.
Any consideration of complexity needs to be complete with an adequate explanation of categorisations. The
process will be applicable across the entire development lifecycle and should be scalable. It is to be expected
that there will be overlap between definitions due to a combination of minor shortfalls in the model and
observer perceptions. It is for this reason that the reasoning behind decisions should be recorded to afford
the best chances of consistency in applying the process in future applications.
As discussed complexity will be viewed from the perspective of satisfying requirements, whether technical
or project management orientated, and will form a part of the framework. There will be interactions between
complexity types and it is highly likely that the nature of the complexity will evolve over the development
lifecycle. We shall call this the Complexity Profile. Complexity assessment will have the potential to become
more detailed as the requirements are defined.
It can be reasoned that a pre-requirement, capturing the ‘business mission’ [28], can only be assessed at a
very high level while detailed system requirements can be decomposed to much greater detail. It is important
that the treatment of complexity can be targeted where it will represent the greatest value, as ambiguity
reduces between the derivation of stakeholder requirements and the final validation of the system [28]. This
can be achieved by considering assessment against either the development’s WBS or Product Breakdown
Structure (PBS), depending upon which is used. Lastly there should be a comparison between actual and
intended outcomes and feedback of learning to improve decision making in both the current and future
development processes. It should be borne in mind that the objective of the assessment of complexity is to
inform future decision making in order to enhance the likelihood of success. Section 3 details the use of CSFs
as an established technique to do this by reducing either the probability or impact of complexity related
events. Both complexity assessment and the assigning of CSFs can be viewed as pre-planning activities and
should be reviewed periodically for applicability and appropriateness.
2.1.1. Complexity themes
Using a combination of the methodologies described within the literature survey and Section 1.4, a typical
technical development can be considered to comprise a number of themes. These are captured under the
three broad headings of internal interfaces, external interfaces and system development. The themes are
designed to provide breadth of coverage over all areas pertinent to system development. Some themes, such
as stakeholders and regulatory interfaces could have been combined but it was felt that the identification of
independent themes would prompt more detailed analysis in areas that are particularly influential. This also
led to the apportioning of a third of the themes to the system development itself, an area underrepresented
in other methodologies such as those from 5DPM and the Helmsman Institute [32]. This ensures that the
assessment is more balanced overall and specifically in areas such as satisfying requirements and completing
scope as a part of technical development.
An important consideration is the project environment within which the development is undertaken. There
will be constraints imposed upon the technical development that will, to varying degrees, be out with the
control of the development management. The organisation type, as discussed in Section 1.4, will be a
significant factor but others include the traditional business-level requirements of cost, schedule and scope
[32]. Organisational culture and business-level governance, such Stage Gate type processes [29], will also be
influential in shaping the technical development. The investment constraints and funding profiles, as
elevated to one of the 5DPM dimensions [31], must also be considered. The remaining themes used in the
framework generally follow the commonly used themes of people, processes and technology [42]. These are
used in business architecture and can be decomposed further to better suit the purposes of the complexity
assessment.

January 2017
Page 10 of 69
People can be decomposed into the internal organisation, i.e. the development team and those external to
it. External elements include contracting organisations, regulatory interfaces and all other remaining
stakeholder groups that contribute to the development. The properties of the development process are
important in terms of detail and number of interfaces. Lastly the technological aspect can be thought of as
the nature of technology that the system will comprise, as well as the integration of the technology (internal
interfaces) and external interfaces, i.e. how the system integrates with other systems. The nine Complexity
Themes used in the framework are listed below. These include high-level considerations that may impact the
analysis.
1. Internal factors
a. (Project) environmental constraints – external factors (inputs and outputs only), schedule
(such as pace and phased handover requirements), the imposed funding profile;
b. Development process – how prescriptive and many interfaces;
c. Internal organisation – ranging from few disciplines within same organization to many and
organisation type;
2. External factors
a. Contractual management – ranging from a few simple relationships to many relationships
with differing contractual terms;
b. Stakeholders - relating to system definition and validation;
c. Regulatory interfaces – how many and how influential they are on the development;
3. System development
a. External (system) interfaces – the degree to which system impacts or depends upon
external systems (from none to part of system of systems);
b. Technology – system requirements definition and verification relating to technology type
(low to very high tech and low to highly novel);
c. Internal (system) interfaces – the level of internal integration required between sub-
systems ranging from few simple relationships to many complicated relationships.
2.1.2. Complexity Criteria
Complexity Criteria can be assessed against each of these themes. The twin themes of uncertainty and
ambiguity will dominate the early phases of any project. Though they are often classically excluded from the
definition of system complexity they have such a profound effect on complexity management and will be
elevated in importance. Rather than be seen as sub-sets of complexity they will sit alongside the other
criteria. Uncertainty and ambiguity will naturally diminish under normal conditions as requirements are
defined, verified and validated.
Conversely emergence and nonlinearity will tend to increase over time as requirements are progressively
defined, implemented, verified and validated. They will reach their zenith as the system approaches
handover when even small changes will have a significant impact. The final criterion is that of Program-size
Complexity. This is essentially the quantity of information that represents the system development.
Techniques to manage this include expenditure of resource and effort (personnel and/or computer-tooling)
and the use of modelling techniques. The system development Complexity Criteria are:
1. Uncertainty – likelihood of unexpected events occurring, e.g. stakeholder requirements change
significantly during system early development or assumptions are found to be incorrect;
2. Ambiguity – incompleteness of knowledge about functional variables [25], i.e., a lack of information
upon which to make decisions. An example may be that stakeholder requirements are undefined in
a particular area of system development leading to assumptions being made or to delays in the
development schedule while the stakeholder requirements are defined;
3. Emergence – how change to system configuration leads to unexpected behaviours and interactions
and re-evaluation of derived system requirements. High Emergence is defined as unexpected

January 2017
Page 11 of 69
behaviours and interactions in many other components or sub-systems;
4. Non-linearity – the effect of a single small change to system requirements on other system
requirements. A non-linear relationship is one where a change results in a disproportionate change
in affected components and sub-systems. High Non-linearity is where a small change results in a large
impact elsewhere;
5. Program-size Complexity – relates to the minimum amount of information required to describe a
process, system or organisational requirements. Manifests itself in shortfalls in modelling and a
general low level of understanding in the system or its development even when uncertainty and
ambiguity is low. High Program-size Complexity requires a high level of effort to model and to
maintain the model. Examples where this may be manifested are requirements management,
configuration management and scheduling.
In summary, uncertainty and ambiguity dominate early in the development lifecycle when their major effect
is lots of low impact changes. Later in the development lifecycle there are less, high impact changes generally
brought about by emergent and non-linear behaviours. Program-line complexity impacts the ability to model,
understand and control these changes, with higher levels of this criteria making schedules and requirements
progressively more difficult to manage.
2.1.3. Complexity Matrix
The Complexity Themes and Complexity Criteria can be presented in matrix form. A scoring system offering
five choices, from very low to very high, has been chosen. Areas to be considered (considerations) for each
of the themes has been chosen from the literature survey [4] and Section 1.5. These looked at sources such
as the Helmsman Institute [16] and NTCP Framework [43]. Considerations provide guidance during
application of the assessment and aid in the assessment of appropriate areas of the WBS against each theme.
Risk has been omitted as a consideration from any of the themes as it is best understood only after complexity
assessment has been undertaken. In this way the assessment will differ from its peers who tend to consider
risk as an input into an assessment rather than an activity that is informed by an assessment’s results.
Considerations can be seen against the themes within the Complexity Matrix as shown in Figure 7.
The most obvious considerations for Project Environmental Constraints are those of imposed cost, schedule
and scope. The funding profile that may constrain the development schedule, as well as overall cost, and
introduce additional assumptions in lieu due to sub-optimal activity sequencing. Components of schedule
include pace, where the development schedule may need to be accelerated to meet external milestones, and
phased handover requirements. There may be additional activities and schedule implications due to project
governance. Other constraints include those relating to procurement and contract usage which may impose
constraints on who contracts are let to and how contracted activities are managed. There may be stipulations
with regards to the technology that is used, for example technologies developed in-house. Lastly,
organisational culture and politics can have a dramatic effect in areas such as communication and decision
making.
Tailoring of the development process to achieve the correct balance of governance and rigour against agility
and flexibility is an important consideration as is facilitation of integration between the technical disciplines.
The use of technology in areas such as requirements management, configuration control and safety case
management will influence not only the development process but also the organisation. The organisation
will strongly influence lines of communication. It is assumed that the fundamental type of organisation
structure is decided early in a project lifecycle. It is generally a difficult undertaking to change it in its entirety
as it is often dictated by the parent organisation undertaking the project. However, it is expected that it may
be influenced by the findings of the Complexity Assessment. Traditional hierarchical structures allow ease of
governance while decentralised network-type structures offer better flexibility [30]. Other aspects include
the number of individual disciplines and their particular experience with regards the development type, the
technology and integration challenges.

January 2017
Page 12 of 69
Figure 7. Complexity Matrix.

January 2017
Page 13 of 69
The internal organisation will interact with contracted development organisations and will be required to
manage the transfer of information (development inputs and outputs) and the interfaces between areas of
scope to ensure there are no omissions or inconsistencies. The development organisation will be shaped by
its place in the wider organisation and particularly how it interfaces with the project organisation. This will
dictate whether the responsibilities for areas such as resource allocation, contract procurement, cost
estimation and scheduling are within the remit of the development organisation and the level of autonomy
it holds.
Other factors include geographical location of disciplines and other interfacing organisations. Contractual
management will of course depend on how much of the development is actually contracted out versus
activities that remain within the internal organisation. The reasons for contracting out of parts of the
technical development will vary, for reasons such as low internal resource or capability or where particularly
specialist or technology specific support is required. The nature of the relationship between the development
organisation and contract organisations will be dependent on the composition of the project organisation.
Responsibilities for actual contract management, including processing of contract variations and scope may
be outside the scope of the development organisation in that they may manage technical interfaces only.
Conversely this may all be within the remit of the development organisation. Again the division between
scope being undertaken by internal resource and that contracted out should be considered against the
outputs of the Complexity Assessment. The identification of significant areas of concern may initiate a
reconsideration of contracting strategy or else prompt the establishment of measures to better manage it
such as rigorous pre-qualification of suppliers. This is an example of a CSF that was introduced as a concept
in Section 1.1 and is subject to more discussion in Section 3.
Stakeholders can be thought of as the direct source of all high-level requirements. They inevitably have a
strong influence on requirements as they are derived in greater detail throughout the development lifecycle.
Factors that will affect stakeholder management include the number and alignment of stakeholders. Non-
alignment of stakeholders’ needs leads to trade-off of requirements to gain the appropriate approvals. Large
numbers of conflicting requirements will require careful management and introduces the risk for conflict and
delays. Other factors include location and availability of access to stakeholders. Incomplete stakeholder
identification may precipitate later changes to requirements and scope with potentially significant impact.
Long development schedules can increase the likelihood of changes to important stakeholders which can
lead to changing requirements also.
Regulatory bodies are important stakeholders which can have a tremendous impact on system requirements.
Technical developments may have weak interfaces, with only one or two regulators, or there may be strong
interfaces, with many regulators. Changes in legislation and regulations may occur at any time due to events
outside the control of the organisation and regulatory regimes may vary greatly between geographical areas.
The development of a system for use in several countries can introduce much complexity surrounding the
management of potentially conflicting regulatory requirements.
External interfaces are systems that interface with the system of interest. There may be few interactions of
a purely transactional nature with other systems or many interfaces. A system being developed as a part of
a system of systems [28] will invoke the greatest management effort. Interfaces may be not properly
understood or may evolve over time beyond the control of the development organisation. Technology is
simply how ‘novel’ and ‘high-tech’ [43] the technology that has been chosen and dictated by the
requirements is. Novelty and high-technology levels, as described by Frank et al [43], generally require highly
iterative development methodologies and high levels of expertise. Short development schedules using such
technology require greater effort, specialised systems engineering techniques and incur disproportionate
level of risk. System integration represents the ‘synthesis of a set of system elements into a realised system
that satisfies system requirements, architecture and design’ [28]. There are likely to be many interfaces that
need to be managed through integration, verification and validation activities. There are techniques to

January 2017
Page 14 of 69
manage this complexity, including early integration planning. Examples of other factors include the
integration of system elements from other contracted organisations and the careful management of
stakeholders to minimise ‘scoop creep’ [44].
Examples of Complexity Matrices, completed for the case studies, can be found in Appendices O and Q.
2.2. How, where and when to assess
Projects invariably utilise a WBS [45] to organise activities prior to cost estimation and scheduling activities.
The assessment considers each Complexity Theme against the five Complexity Criteria at each node of the
WBS on a particular level. Scores are assigned (very low, low, medium, high and very high) at each
intersection of the matrix. As the WBS increases in detail so does the effort required to undertake the
assessment. It may be justifiable to limit the assessment to relevant portions of the WBS or choose a high-
level within the WBS to restrict the nodes to be considered.
The complexity assessments should be undertaken against the portion of the WBS that describes the
technical development aspects. The WBS, including that describing technical development, will be subject to
refinement over time. As such any early assessment will be at a high-level on the WBS and necessarily at a
corresponding low-level of overall detail. There will be business-level requirements at the offset and other
factors, such as generic organisational processes and the existing regulatory landscape. Additionally, early
planning may make assumptions with regards to the organisation and contracting mechanisms. This pre-
requirements early assessment will identify areas of potential concern to enable early planning through the
identification of high-level CSFs.
As the project and the development progresses the WBS will be defined level by level with early development
phases likely to receive the most detail. The example of a WBS shown in Figure 8 has technical development
activities below several of the high-level nodes. In this example requirements definition would receive more
attention than test and verification activities due to the dependencies between the two and the inherent
difficulty in defining later activities because of this. Complexity assessments would be undertaken as a
minimum at the end of development phases and would serve to inform the planning of subsequent phases
and as an important input into Stage Gate type governance reviews.
Long project phases may demand interim complexity assessments to catch any significant changes during the
phase. Assessments would develop the earlier higher-level reviews to progressively lower levels of the WBS
to identify hotspots of complexity within the development. This technique affords the opportunity to
Figure 8. Sample WBS for an unmanned aerial vehicle (UAV) [46].

January 2017
Page 15 of 69
disregard areas of low complexity and allow expenditure of effort where it needed. For the purposes of this
example WBS ‘control software’ and ‘verification software’ have been identified as having high levels of
complexity. This could prompt the assessment of the next level of the WBS below to determine the particular
node that is of particular concern. This can be repeated as required and, in the most extreme application,
until the lowest level of the WBS has been reached.
2.3. Outputs from complexity assessment
Identification of complexity should be used inform development planning, enabling the establishment of
conducive project environment. An established method is that of CSFs as environmental influences that will
enhance the likelihood of success. They may reduce either the probability or impact of the consequences of
complexity. Both the rationale and the proposed outcome of the putting in place of CSFs should be recorded.
This exercise should also prompt the identification of development risks. Some of these will be residual risks
as a result of putting in place of CSFs.
The putting in place of CSFs will influence the complexity profile. An example of this, developed later within
the case study in Section 8, is that of the development of Boeing’s Dreamliner. Radical changes in Boeing’s
procurement strategy were implemented at the very early stage of the project to improve schedule and costs.
This changed the profile of complexity within the project’s supply chain, introducing ‘’coordination risks’,
with well-publicised results [47]. Using the complexity themes, it may be that complexity within the internal
organisation are merely transferred into contractual management. It is therefore important that the
complexity assessment is repeated in areas where changes are proposed. Indeed, in the example of AREVA’s
design and construction of the Olkiluoto 3 nuclear plant the issues were entirely the reverse of the
Dreamliner and related with AREVA choosing to embark on a first of a kind project alone without the
necessary experience as ‘architect and engineer’,’ without experienced partners’ and without having all the
necessary competences [48]. Repeating the complexity assessment will identify any further areas of
complexity that have resulted from the implementation of CSFs and allow the impact of changes to strategy
to be understood. Additional CSFs can then be put in place or existing CSFs amended as appropriate. The
designation of CSFs is discussed further in Section 3.
2.4. Complexity profile and the interaction of complexity criteria
Complexity is influenced by many factors and is not a static property. It will evolve over the development
lifecycle and its nature may also be transformed by imposed changes. It can be theorised that Complexity
Criteria will interact in specified ways and will follow individual general profiles. Each criterion applies equally
to modelling of the development process through planning and scheduling as it does to the definition of
system elements and their interactions.
Uncertainty relates to the likelihood of
change of development requirements and
it will be the greatest at the beginning
when definition of requirements and scope
is at its lowest. Naturally as definition
increases and the number of assumptions
are reduced, uncertainty is also reduced.
Uncertainty can relate to changes
prompted from sources such as stakeholders, regulators or rework due to iteration or errors. Uncertainty
should reduce as the system is verified, and should diminish to its very lowest level as the system approaches
final validation. Changes, especially those later in the development, will reintroduce uncertainty. Ambiguity
is closely related to uncertainty and relates to the amount of information that is unknown. In the absence of
verified information, it is necessary that assumptions are made to enable aspects of the development to
progress. Without assumptions other activities will be delayed. Ambiguity will reduce as requirements are
Figure 10. The changing nature of the project process [25].

January 2017
Page 16 of 69
defined but as with uncertainty it will
be increased with the introduction of
changes. Early definition activities will
reduce both uncertainty and
ambiguity together.
Two methods of detail with relatively
low levels of uncertainty and
ambiguity include the inclusion of
‘float’ [49] within development
schedules (sometimes known as slack)
and ‘contingency’ [50] within the cost
plan or the inclusion of flexibility within the design to accommodate a range of design definition outcomes
[51]. Emergence, as defined in Section 1.5, will increase throughout the development lifecycle as ambiguity
is reduced. This property is invoked by the interaction of requirements or by the introduction of change in
ways that cannot be anticipated. It specifically relates to the impact and the likelihood of emergent
behaviours arising. The impact of emergence can be managed through the decoupling of development
activities or system elements. Examples of managing emergence are that of removing activities with likely
emergent behaviours from the critical path of a development schedule or controlling the interfaces between
sub-systems through choice of system architecture. Non-linearity also increases as ambiguity is reduced. The
impact of non-linearity can be
managed by identifying such
interfaces and introducing extra
capacity in the affected system
elements to manage potential
future change. This should be
undertaken on the basis of risk and
the extra capacity can be either built
into the changed requirement to
prevent the change being
propagated or those affected by the
non-linear behaviours to absorb the
change with minimal impact. It is
highly desirable to reduce
uncertainty and generally control
the incidences of change as non-
linearity and emergence increase.
Considering this from the point of
view of requirements and scope it
can be reasoned that Program-size
Complexity of the development will
begin and end low. Requirements
will start as those necessary for the
business or operational need and
will be relatively few and at a low
level of detail. Requirement
Program- size complexity will rise as
requirements are defined,
Conceptofoperations
High-levelrequirements
Detailedrequired
High-leveldesign
Detaileddesign
Implementation
Integrationandtesting
Sub-systemverification
Systemverification
Operations&maintenance
Figure 11. Complexity Profiles over the development lifecycle with
no major changes and an incidence of major change during sub-system
verification (bottom).
Point of major change
Figure 9. Level of organisational effort over time [52].

January 2017
Page 17 of 69
increasing in both number and detail. Conversely this property will reduce as requirements are verified until
the business and operational level requirements are finally validated. Scope is similarly represented via the
WBS and activity scheduling. Again these begin at a low-level of detail until the development is fully defined
and planned. As activities (scope) is completed the Program-size Complexity again reduces as the number of
activities to complete and their interactions reduces. This is broadly in line with organisational effort [56] as
can be seen in Figure 9. Changes will again increase Program-size Complexity with the example of previously
verified requirements needing to be reanalysed and adapted to accommodate a late change. Program-size
Complexity is managed through the application of organisational or computing resource and/or modelling
techniques. The magnitude and impact of Program-size Complexity over the design definition phases is
shown in Figure 10 [25]. Pich et al propose that not only does the complexity of information increase over
time but that its relative impact of information content on outcomes diminishes over time. This assertion
chimes with that of INCOSE who propose that approximately 70% of committed lifecycle costs are due to
concept design against 95% [28].
Figure 11 suggest how the profile of a typical technical development may be represented over its lifecycle
with an indicative measure of the magnitude of the particular complexity criteria against time with the
development phase also shown. The first graph shows no major changes while the second graph shows a
major change that is brought about during sub-system verification. This is manifested by an increase in
program-line complexity, uncertainty and ambiguity. Development phasing has been taken from INCOSE as
per Figure 4 on page 5. Such late changes present issues are there are now additional activities to manage
and potentially a large number of system elements, previously verified, need either to be completely re-
evaluated or re-verified against the new requirements. Uncertainty and ambiguity are re-introduced into
system elements that had previously been defined and built or constructed. The impact of non-linearity and
emergence is high due to the timing of the change and the number and nature of system dependencies.
3. Critical Project Success factors and their application in system development
Critical Success Factors were first proposed by D. Ronald Daniel in the 1960’s, after which they were further
developed by John F. Rockart of the Sloan School of Management [5]. Their application was primarily
intended within the areas of business strategy but have also been adopted in project management as success
factors [45][53]. It is the intention that the selection and implementation of CSFs should put in place an
environment more conducive with a successful outcome.
3.1. Selection of success factors from literature
Two mistakes that should be avoided are those of confusing a CSF with a requirement and assigning the CSF
as too high a level. The latter make the CSF too general for useful application. Requirements are distinctly
different from CSF and though their satisfaction is as at least as important, their management is handled
elsewhere. Examples include particularly important stakeholder requirements being associated with
‘Measures of Effectiveness’ (MoE) and system requirements with ‘Key Performance Parameters’ (TPP) of the
system of interest [54]. These will be discussed more fully in a later section but as performance measurement
metrics.
A CSF that is described at too high-level across all activities can be viewed as pre-requisites for any technical
development, project or business endeavour. Examples of high-level CSFs are provided by the APM Body of
Knowledge [45]:
 Defining clear goals and objectives;
 Maintaining a focus on business value;
 Implementing a proper governance structure;
 Ensuring senior management commitment;
 Providing timely and clear communication.

January 2017
Page 18 of 69
While these obviously worthy objectives, they apply to every project that the author has worked on.
Furthermore, they give no indication of how they might be judged as being satisfied. It is therefore important
that the CSFs do not merely represent best practice but apply to the demands of the particular technical
development as indicated by complexity assessment. As such, the CSFs selected for inclusion in the
framework will need to be supplemented with enough detail for implementation and, if possible,
measurement.
I have selected 141 CSFs for inclusion in the framework from a variety of sources, including APM [7], Pinto
and Slevin [52], Dvir et al [55], Chow and Cai [56], Ragatz et al [57], Koutsikouri and Dainty [58] and Fortune
and White [59. These are in turn divided amongst the Complexity Themes. Some of these CSFs are repeated
in more than one Complexity Theme. Ultimately each will be need to be developed with the appropriate
detail if chosen in practice and adopted for use. For example, ‘support from senior management’ should
include whose support should be in place and when and what this support will achieve. It should be targeted
against particular activities and should recognise the influence that development phasing may have on the
nature of management support that is required. This who, what, how and why approach can be applied to
most items on the list. The full list of CSFs for the framework has been grouped into the applicable Complexity
Themes and can be found in Appendix C.
There are a number of important concepts that should be borne in mind when applying CSFs. Merely stating
development best practice will dilute the effectiveness of the identification of CSFs and instead only
particularly areas of the process should be subject to analysis. As such general process improvement should
be addressed elsewhere. In the spirit of the incorporation of the principles of the Demming cycle, and the
resulting continual improvement, the proposed and the actual effect of the CSF should be recorded. This will
enable the effectiveness of individual CSFs and allow them to be catalogued within a toolbox of CSFs for
future use. To facilitate in the analysis of the effectiveness of CSF they should have performance measure
associated with them where possible.
3.2. Verification of success factors through questionnaire
Potential CSFs identified in the literature were evaluated for their potential utility in the framework via a
questionnaire. The questionnaire used the criterion of perceived impact on the likelihood of project success.
The questionnaire was posted online using the SurveyMonkey application [60] and a total of 122 responses
were achieved by requests via email and Linkedin social media. An overview showing the source of replies
can be seen in Appendix D. Respondents were a varied mixture of engineering and project management
professionals. Respondents were asked to rank the CSFs listed in Appendix C according to their potential
influence on a project into five categories from very low to very high. Any CSFs not considered relevant or
without an impact were removed from the list as being non-applicable. The questionnaire also gave
respondents the opportunity to identify other CSFs that the literature survey and subsequent analysis had
missed. The questionnaire and explanatory text is contained with Appendix E. The unprocessed results, per
respondent, are contained in Appendix R. Processing of the CSFs was undertaken and they are categorised in
Appendix F according to their scores:
 4.0 and above - shaded in green (high to very high influence);
 3.5 and above - shaded in yellow (upper scoring of medium to high influence);
 3.0 and above - shaded in orange (low scoring of medium to high influence);
 Below 3.0 - no shading and red type (low to medium influence).
These categories could then be filtered, sorted and compared across age, role description and industry.
Further processing was then undertaken to aggregate relevant age and role group. Data was disregarded
where this was not possible for particular role and industry groups. This ensured that data groups were
statistically significant, allowing the data from the selected groups to be meaningfully compared to the full
dataset. Information relating to the size of the aggregated datasets can be found in Appendix G.

January 2017
Page 19 of 69
3.2.1. Overview
The findings were analysed across the full dataset and then these results compared to the various categories
sorted by age, role description and industry. In this way trends within the sub-groups could be identified. It
may be surmised that CSFs and performance measures for different industries will be different and as such
complexity managed differently across different domains. Also, the influence of age and role description on
CSFs and performance measures may skew complexity management due to the biases inherent with them.
Examples being project management professionals placing greater importance on established project
management techniques such as Earned Value Management while engineering personnel may place a
greater emphasis on the management of requirements. Identification of such biases will aid in the
composition of teams determining how complexity management will be undertaken. Known biases can then
be predicted and defended against. Comparisons were made between the filtered data and the full dataset.
This was done for the high to very high (shaded green) CSFs and then for the top three CSFs in each category.
83% of the respondents were from the age categories of ‘35 to 44’ (25%), ‘45 to 54’ (27%) and ‘55 to 64’
(31%). The results were consolidated from seven into five age categories. Consolidation of results outside of
the popular categories formed ‘up to 35’ and ‘over 65’ categories. This ensured sample sizes were statistically
significant. ‘Up to 35’ was the smallest data group with just below 7% of responses.
Role descriptions were similarly consolidated to ensure that samples for each category were sufficiently
sized. The data from 12 respondents was disregarded as not fitting with any one of the three aggregated
high-level role description that were chosen, reducing the dataset to 110. The groups derived from across
full dataset were ‘senior personnel’ (41%), ‘project personnel’ (40%) and ‘engineering personnel’ (19%).
Industry categories with the largest sample sizes were considered for analysis. Any groups below 10 in
number were disregarded resulting in a total of 83 respondents. Those chosen based on sample size were
‘construction’ (20%), ‘decommissioning’ (16%), ‘defence’ (16%), ‘energy generation’ (18%), ‘oil and gas’ (12%)
and ‘professional services’ (18%). Obviously there is some overlap between these industries and particularly
between ‘construction’ and ‘professional services’, which are generic terms that may relate to undertaking
of particular activities across most industries. The chosen data group sizes were generally of similar in size,
with ‘oil and gas’ being the smallest with 10 responses.
Further work, through a larger sample size, could be undertaken to increase the number of industries that
can be analysed. This would also improve the general confidence in the findings across age, role description
and industry. Furthermore, biases could be more completely determined across individual role descriptions
rather than consolidated into the groups that were chosen.
3.2.2. Full dataset
The summarised results, as discussed here, are contained within Appendix F. This section will discuss the top
three CSFs for each complexity theme and the trends that were identified by the author from this data. The
analysis was across the data from all 122 respondents.
Imposed project constraints
 Clear realistic project objectives with an average (mean) score of 4.51 and 68 of 118 (58%) rating it
very high;
 Adequate budget with an average (mean) score of 4.39 and 62 of 120 (52%) rating it very high;
 Composition of project team in terms of experience and capability with an average (mean) score of
4.38 and 57 of 120 (48%) rating it very high.
In general planning and general business and governance processes appeared a lot less important than a
good organisation, certain project management processes (change and risk management) and the overall
viability of the development. Strong business case/sound basis for project was notable for having the second-
highest number of very high ratings but equal tenth-highest number of low ratings amongst the 4.0 and above
CSFs. This gave it a relatively lowly seventh most influential overall. This demonstrates how the perceived

January 2017
Page 20 of 69
importance of a business varies amongst respondents. Project management respondent’s generally assigned
this very high-importance, in line with the teachings of project management methodologies, while other
groups gave it a much lower score.
Planning related CSFs were ranked as a high ‘medium to high’ influence or a low ‘high to very high’ influence
which was unexpected considering the relative importance of planning within project and engineering
management. A project cancellation process was assigned a ‘low to medium’ influence so should be
discounted from the list of CSFs. Evidently the cancellation of an ill-conceived or no-longer-required project
was considered either not relevant or of a low overall influence. None of the comments provided CSFs not
already included here or within another Complexity Theme’.
Technical development processes
 Critical activities are identified with an average (mean) score of 4.47 and 63 of 115 (55%) rating it
very high;
 Clear realistic development objectives with an average (mean) score of 4.32 and 53 of 116 (46%)
rating it very high;
 A well understood and mature design review process is in place with an average (mean) score of 4.27
and 56 of 118 (47%) rating it very high.
Unsurprisingly the general trends of the first Complexity Theme were evident in this one. Identification of
areas of risk and criticality ranked higher than general planning. Common Systems Engineering techniques
such as ‘test early, test often’, ‘modelling and prototyping’ and ‘standardisation and/or modularisation’ were
assigned a relatively low importance overall. Examples of additional comments include the use of ‘state of
the art’ development processes and ‘clearly defined and mature functional requirements’.
Organisational
 Good leadership with an average (mean) score of 4.62 and 72 of 115 (63%) rating it very high;
 Transparent definition of responsibilities with an average (mean) score of 4.26 and 51 of 115 (44%)
 Degree of collaboration with an average (mean) score of 4.26 and 47 of 114 (41%) rating it very high.
Leadership was by some way the most important influence. Co-location of teams was not thought to be
important which may be reflected by the global nature of modern projects and the availability of better
technologies to allow collaboration. Surprisingly ‘competence in technology and technology management’
and ‘domain-specific know-how’ were not considered as ‘high to very high’ influences.
Contractual management
 Clearly understood contractual interfaces with an average (mean) score of 4.45 and 64 of 114 (56%)
 Good performance by suppliers/contractors/consultants with an average (mean) score of 4.43 and
58 of 114 (51%) rating it very high;
 Effective monitoring/control with an average (mean) score of 4.38 and 54 of 114 (47%) rating it very
high.
The ranking of influences did not show any unexpected trends which were in essence the principles of
selecting a competent supplier, ensuring the contract was clearly understood and managing it well.
Stakeholder management
 Client/user acceptance with an average (mean) score of 4.58 and 74 of 113 (65%) rating it very high;
 Decisions are agreed and documented with an average (mean) score of 4.48 and 64 of 113 (57%)
 User/client involvement with an average (mean) score of 4.44 and 64 of 113 (57%) rating it very high.

January 2017
Page 21 of 69
Stakeholder management reinforced the need for early and effective communication. The representation of
the full system lifecycle was not thought to be particularly important though the involvement of operators
was considered as a relatively low in ‘high to very high’ category. The conclusion here is that the eventual
decommissioning of a system is often of low concern during its design and implementation.
A respondent’s comment that may be worthy of further investigation is that of ‘no informal communications
under any circumstances and record all meetings’. The use of formal versus informal communication is an
area that is rarely discussed. Informal communication infers frequent and relaxed use of the mediums of
telephone calls and email, which suggests trust and collaboration. The inverse is contractually driven, with
an administrative burden and an impact on factors such as frequency and content.
External interface management
 Clear communication is established with an average (mean) score of 4.41 and 56 of 112 (50%) rating
it very high;
 Clearly identified and understood external interfaces with an average (mean) score of 4.33 and 55 of
113 (49%) rating it very high;
 Defined process for managing external interfaces with an average (mean) score of 3.96 and 29 of 115
(26%) rating it very high.
This theme mirrored the need for structured and effective communication as discussed in the previous
theme. Only two of the six were rated above 4.0.
Regulatory interface management
 Clearly identified and understood interfaces with an average (mean) score of 4.45 and 59 of 111 (53%)
 Clear lines of communication with regulators with an average (mean) score of 4.36 and 52 of 110
(47%) rating it very high;
 Good relationship with regulators with an average (mean) score of 4.28 and 48 of 111 (43%) rating it
very high.
All categories within this theme were assigned a relatively high score and even more so than other external
interfaces. This is obviously down to the criticality of the regulator’s role in many projects.
Technology development management
 Well-defined standards up front with an average (mean) score of 4.04 and 31 of 111 (28%) rating it
very high;
 Proven/familiar technology with an average (mean) score of 4.01 and 38 of 111 (34%) rating it very
high;
 Pursuing a simple as possible design with an average (mean) score of 3.98 and 38 of 111 (34%) rating
it very high; 3.98 and 38 of 111 rating it very high.
The conclusion when considering the highest scoring CSFs within this theme was the avoidance of
unnecessary complexity while using established and thoroughly documented design standards. This is often
not possible and many projects have elements of research and development within them. These were
generally rated a lot lower than the other themes with approximately half the number of very highs assigned.
System integration management
 Well-defined standards up front with an average (mean) score of 4.17 and 37 of 112 (33%) rating it
very high;
 Pursuing a simple as possible design with an average (mean) score of 4.01 and 36 of 113 (32%) rating
it very high;
 Proven/familiar technology with an average (mean) score of 3.98 and 30 of 111 (27%) rating it very
high.

January 2017
Page 22 of 69
The scoring was similar to that within technology management. Only two of the eight were rated above 4.0.
Again the numbers of very highs assigned were around half that as other themes.
3.2.3. By age
Results filtered by age can be seen in Appendix H. Sample sizes per any age group were at best a third the
size of the whole dataset. Instead of analysing absolute ratings the top rated CSFs for each theme were
compared against the full dataset for consistency.
Imposed project constraints - Up to 35 generally scored lower with a lot fewer ‘high to very high’ category
CSF. This could be an anomaly as a consequence of the relatively low sample size. ‘35 to 45’ were
representative of the overall rankings. Age ‘55 to 64’ and ‘above 64’ advocated ‘strong project
sponsor/champion’ and ‘competent and qualified project manager’ respectively. Both these age groups rated
‘strong business case/sound basis for project’ which perhaps reflects their position and seniority within their
respective organisations, introducing a related bias.
Technical development processes - The ‘below 35 age group assigned more importance to planning in areas
of criticality and the use of past experience, while the ‘above 64’ age group gave more importance to planning
and responsiveness. Other age groups were broadly representative of the trends within the full dataset.
Organisational - Findings across all age groups were closely in agreement with the full dataset and especially
near the top ranked CSFs.
Contractual management - Findings across all age groups were closely in agreement with the full dataset and
especially near the top ranked CSFs though the position of ‘rigorous pre-qualification process’ varied greatly
across age ranges.
Stakeholder management - Findings across all age groups were closely in agreement with the full dataset
though the ’35 to 44’ age group assigned a lot greater importance to the managing of expectations and the
representation of operators.
External interface management - Findings across all age groups were closely in agreement with the full
dataset
Regulatory interface management - Findings across all age groups were closely in agreement with the full
dataset
Technology development management - Findings across all age groups were closely in agreement with the
full dataset though the ‘up to 35’ age group assigned high importance to ‘continuous improvement process
for products’.
System integration management - This theme was very similar to technology and was again closely in
agreement with the full dataset with the ‘up to 35’ age group assigning high importance to ‘continuous
improvement process for products’.
3.2.4. By role
Results filtered by role can be seen in Appendix I. Similarly, to by age these sub-datasets were a fraction of
the full dataset at less than half the size. The sub-data sets were compared against the full dataset results.
Imposed project constraints - The findings were closely in agreement with the full dataset across the role
descriptions with the exception of an understandable bias towards a single CSF in each role. Upper
management advocated a ‘strong business case/sound basis for project’. Project personnel ‘ranked most
highly the use of a ‘competent and qualified project manager. ‘Support from senior management’ was the
third most important CSF for engineering personnel which perhaps indicates its importance gained from
previous experience.
Technical development processes - Findings across all role descriptions were closely in agreement with the
full dataset across senior management and project personnel but perhaps understandably engineering

January 2017
Page 23 of 69
personnel had a very different perspective. The following CSFs, were not present in the other role
descriptions. The first two of these were second and third most important while the others descended in
relative influence within the ‘high to very high’ category.
 Enhanced planning is applied against areas of criticality and uncertainty;
 Fast transfer of information;
 Test early, test often philosophy is used during development;
 Past experience of management methodologies and tools is available;
 Correct choice of management methodologies and tools;
 Trouble shooting mechanisms in place.
Organisational - Findings across all role descriptions were closely in agreement with the full dataset with the
exception of the inclusion of’ composition of development team in terms of experience and capability’ within
engineering personnel.
Contractual management, Stakeholder management, External interface management and Regulatory
interface management - Findings across all these themes and across all role descriptions were closely in
agreement with the full dataset
Technology development management - Findings across all age groups were closely in agreement with the
full dataset though engineering personnel rated the CSFs general lower with only ‘test early, test often
philosophy’ achieving the highest ranking.
System integration management - Findings across all age groups were closely in agreement with the full
dataset though engineering personnel rated the ‘modelling and prototyping’ higher than the other role
descriptions in the top three.
3.2.5. By industry
Results filtered by industry can be seen in Appendix J. Sub-datasets were again compared.
Imposed project constraints - The findings were closely in agreement with the full dataset across the role
descriptions with a few notable exceptions within oil and gas. Perhaps as a reflection of an environment
reliant on oil prices and commercial pressures ‘effective change management’ and ‘political stability’ ranked
relatively highly.
Technical development processes - Construction differed significantly with the following CSFs figuring in the
rankings as ‘high to very high’, with responsiveness being the third most important. This reflects the dynamic
nature of a construction environment and requirement to act quickly during the construction phase to
minimise or avoid rework.
 Responsive and flexible process to meet client needs;
 Strong, appropriately detailed and realistic development plan kept up to date;
 Appropriate development planning technique has been chosen;
 Correct choice of management methodologies and tools.
Organisational - Energy generation elevated the ‘re-use knowledge and experience from previous projects’
which is a strong component of the nuclear following events such as Three Mile Island and Chernobyl
[66][67]. Oil and gas elevated the ‘collocation of teams’, perhaps due to the international nature of many of
its projects. Professional services included ‘domain-specific know-how’ and ‘appropriate techniques to aid
identification of organisational dependencies and interfaces’. Otherwise there was commonality between
the industries and most influential CSFs.
Contractual management - With some minor differences findings across industries were closely in agreement
with the full dataset.
Stakeholder management - With some minor differences findings across industries were closely in
agreement with the full dataset.

Integration of technical development within complex project environment

Integration of technical development within complex project environment

Recommended

Recommended

More Related Content

What's hot

What's hot (10)

Similar to Integration of technical development within complex project environment

Similar to Integration of technical development within complex project environment (20)

Recently uploaded

Recently uploaded (20)

Integration of technical development within complex project environment