Preliminary SOA Indicators/Metrics: A starting point for technical
discussion / elaboration
November 5, 2006
John Salasin, Ph.D.
While all errors of omission and commission are those of the author, comments
have been provided to date by:
• Navy Ocean Systems Command (NOSC) and NOSC SSC
• DISA / NCES
• BEA Systems
• Solstice Software
• Federal Chief Information Officers’ Council (CIOC), Architecture and
Infrastructure Committee, Services Subcommittee
• Federal Aviation Administration (FAA)
1 Objectives and Assumptions
To identify measures (characteristics of the application and technology) early
enough in the life cycle to will influence success in later stages (Predictive
1.1 Metrics are collected for a reason, and are based on some underlying
assumptions about a system life cycle. Some of the assumptions underlying
this paper are:
1.1.1 The Predictive Metrics should provide a basis for continual system-level,
Verification & Validation (V&V) that includes assessing both the underlying
technology and the application(s) as development proceeds. The metrics
should be useful in deciding if there is, at any point in its
development/evolution, a high probability that the system will meet its
requirements – and a minimal risk that it won’t. Since we are emphasizing
technical aspects of the system and their match to an application and its
organizational context, the paper addresses only part of what V&V
requires. It does not discuss, for example, traditional requirements review /
validation – although this is being increasingly conducted using
prototyping technology, which is discussed.
1.1.2 Since an estimated 2/3 of a systems’ cost comes after it’s initial
deployment, in maintenance or evolution, many of the metrics represent
factors that are likely to influence Total Cost of Ownership (TCO). Thus,
for example, one of the central principles of an SOA-based system is that
the services and their orchestration map directly to business processes.
Since many organizations undertake “Business Process Reengineering”
every 12-18 months, the ease of reflecting a business process change in
the SOA can have a major and continuing cost impact. A major factor
impacting the ease of synchronizing the business process with the SOA is
the notation used to express each. Ideally, they would be the same or
could be automatically mapped to each other – representing close to zero
cost. If the translation has to be done manually, costs are likely to increase
dramatically and translation errors to be introduced. Similarly, the “stack”
of notations, ranging from the highest level service architecture to
executable code, needs to be consistent to allow automated tool
assistance in refinement or in checking the correctness of manual
1.1.3 Since we are considering system evolution, including initial development,
as a continuous process, rather than a discrete sequence of products /
reviews (e.g., requirement documentation and review, architecture
documentation and review, code documentation and review,…) the
metrics should be collected when available to support decisions, rather
than at pre-defined points. For example, metrics related to data should be
collected when there are significant changes to the data sources being
used or the structure or semantics of one or more data sources.
1.1.4 Models are the “global invariants” of development. They must remain
consistent, yet be adapted to express different levels of refinement and
accuracy as design/implementation decisions are made and additional
data collected. Different sets of metrics from families of similar models at
different levels are needed to illuminate different views of the system –
from effectiveness and efficiency at supporting the business mission (e.g.,
impact on business performance versus resources required) to technical
concerns (e.g., response time, throughput, computer and communication
1.1.5 Organizational characteristics will influence both the models employed
and the approaches used for building an SOA-based system. Much as the
Capability Maturity Model (CMM) and Integrated Capability Maturity Model
(CMMI) measure organizational process (management) capabilities,
different organizations, programs and systems differ in the sophistication
of the approaches they use for sharing information, engineering services,
or increasing their value. Models might reflect organizational
characteristics according to the “Mode of Delivery” they support (“Modes
of Delivery” briefing, August 8, 2006, Kshemendra Paul, Chief Enterprise
Architect, US Department of Justice). These include:
o Sharing Information
1. About what aspects of technology?
2. Over what domain of consumers – from a single program to the
3. Using what mechanisms?
o Enhancing service value modes:
4. Service management -- what is managed?
5. Service adoption – how is information to improve a service
obtained and used?
6. Service Model – what methods used to commoditize a service
to help offset operation cost?
o Service provision level modes:
7. Service efficiency – how can existing artifacts, including
designs, legacy systems and COTS, be used to reduce costs?
8. Engineering practice – Can the organization design a better
service by moving away from ad-hoc, “one-off” designs to using
more standardized and formal patterns, models or frameworks?
2 Predictive Metrics -- Definition
The aim is to identify “Predictive Metrics” -- measures (characteristics of the
application and technology) that can be made early in the life cycle based on
(hopefully explicit) models of the system (technology and applications(s)) and will
influence success in later stages. Hopefully, these will suggest likely problems /
opportunities at future stages of development / deployment and will inform
decisions about corrective action.
A simple, hypothetical example may help clarify what we mean. Assume that our
initial concept is to avoid the cost and politics required to reconcile a number of
ontologies used in separate “stovepipe” systems that are to be coordinated in an
SOA-based EA system. We plan, instead, to use the SOA’s Enterprise Service
Bus’ transformation capabilities to reconcile the data at run-time. Metrics that
might be useful in gauging the effects of this (conceptual) design decision on
system performance and development/evolution cost include:
• The overhead required for translation as a function of the number of
separate ontologies that must be reconciled. A high value could indicate
degradation in response time or throughput (transactions/minute) as
function of the number of independent ontologies (or number of data
elements) that must inter-operate.
• The percent of transformations required by the SOA system that can be
expressed in simple scripting language (e.g., by business managers vs.
programmers). A small number could indicate the need for extensive
programming effort using more highly skilled individuals, increasing
development cost and time. It could also indicate increased difficulty in
evolving the system to account for changes in the data.
It seems evident that the availability of such metrics could inform design
decisions so as to avoid potential future problems.
3 Life Cycle Stages
Since we do not wish to tie the metrics to any specific life cycle methodology, we
discuss the metrics in terms of four (4) types of effort – which may overlap,
iterate, or be conducted in parallel. We characterize these stages as:
3.1 Early stage – concept development, obtaining management support.
At this point, the functions of the system and other organizations,
with which it must interact are specified in general. Based on the
functionality, we can estimate the scope of the services provided in
terms of the number and types of consumers – from a single program
to the public at-large.
We can make some initial estimates of which (and how many) organizational
units are performing similar functions with similar (or the same) data –
allowing us to develop plans for how technology, information, and costs might
be shared. Based on these organizations’ characteristics with respect to the
formality of their processes for specifying information and engineering,
adapting and managing services, we can select mechanisms to foster the
adoption of the services (or components of the service) by these other
organizational units. This should allow us to make initial decisions about how
efficiently the system can be constructed (e.g., what legacy or COTS artifacts
are available) and whether the system can or should be part of an overall
service framework that is based on a model driven architecture and standard
design patterns, versus an ad-hoc, “one-off” effort.
The organizational elements that are sites for initial implementation /
demonstration have developed (at least preliminary) models of the system’s
value in terms of:
3.1.1 Mission and Business Results that capture the outcomes that agencies
seek to achieve and begin to identify quantitative measures of these
3.1.2 Customer Results that capture how well an agency or specific process
within an agency is serving its customers and suggest ways of measuring
these results, e.g., using measures related to:
• Service Coverage;
• Timeliness & Responsiveness with respect to the time to respond to
inquiries and requests and time to deliver products or services, and;
• Service Quality, both using objective measures and based on the
customer’s perspective. This includes factors such as: Service
Accessibility, Customer Benefit, Customer satisfaction levels and
tangible impacts to customers.
• Comprehensiveness, the percent of the customers’ needs that are
satisfied by the service, is an important aspect of quality. Thus, if the
service is intended to provide managers with information about sales,
does it provide information about sales that is restricted to a subset of
products sold or a subset of sales regions, versus providing
information about all sales at whatever level(s) of granularity and
specificity that is desired? Does it provide the information to all
customers who need it?
3.2 Architecture/Construction. A number of SOA components are being
specified and implemented in this stage, including:
• Services, possibly as the composition of lower-level services;
• An architecture (service orchestration within an enterprise or
organization, choreography across enterprises or
• Infrastructure components (e.g., an Enterprise Service Bus or
Portal development tools);
• Data/information and process (Service) reconciliation and
integration/coordination mechanisms and tools, and;
• Inter-service contractual agreements (e.g., SLAs regarding Quality of
Many of the concepts developed in the early stage can be evolved into finer
grained and more accurate quantitative models (e.g., of data, service
interoperability, cost sharing, etc.) FEA Measurement Areas that are most
significant at this stage include:
3.2.1 Processes and Activities intended to capture business results that are the
direct result of the process that an IT initiative supports. These include,
• Business productivity and efficiency,
• Quality in achieving financial measures related to costs of
producing products and services, and costs saved or avoided,
• Cycle time and timeliness to produce products or services with
respect to mission requirements (“time to market”),
• Assessments of management policies and procedures, compliance
with applicable requirements, capabilities in risk mitigation,
knowledge management, and continuous improvement and,
• Security and privacy improvements in accordance with applicable
These measures can be estimated in the Architecture/Construction stage and
validated in the Operations and Evolution stages. The estimates and data can be
compared with previous efforts, or those undertaken using more traditional
approaches, to assess the “speedup” provided by SOA technology.
3.3 Operations. This stage will refine the measurements / models of Processes
and Activities initiated in the Architecture/Construction stage. It can begin to
focus on the FEA measurement area of Technology to capture key elements
of performance that directly relate to the IT
initiative. This includes factors related to, e.g.:
• Technology quality, regarding how well the
technology satisfies functionality or
capability requirements and complies with
• Reliability & Availability, reflected by the
system’s availability to users and system or
• Technology-related costs and costs avoided
through reducing or eliminating IT
redundancies and using COTS or re-using
• Extent to which data or information sharing, standardization, reliability and
quality, and storage capacity are increased.
• Efficiency in terms of response time, interoperability, user accessibility,
and improvement in technical capabilities or characteristics
• Effectiveness as judged by the extent to which users are satisfied with the
relevant application or system, whether it meets user requirements, and its
impact on the performance of the process(es) it enables and the customer
or mission results to which it contributes.
3.4Evolution – All systems must evolve based on changes in
requirements or threats, policy, data availability, and
equipment. In long-lived systems, this cost is estimated at 2/3
of the total lifecycle cost. The ease with which a system can be
modified needs to be considered from the early conceptual
stage. It will be influenced by the, e.g.:
• System design (e.g., modular or layered systems are
easier to modify than monolithic systems since
changes can be localized) and,
• Ways in which the system is represented at various
levels of refinement – representations that support
automated refinement or error checking will make system
modification much easier.
In addition, evolution includes the expansion of a system to include additional
organizational units, functions, ect. The suggestion has been made that we “need
to think strategically, but implement tactically”. Very few organizations are about
to take the business and technical risks of adopting a “big bang” approach –
going home on Friday with the old system in place and returning on Monday,
after IT has spent the weekend working, with the new SOA/EA system in place. A
more feasible approach is to do development tactically – concentrating on a
small unit or collection of functions that needs an immediate IT upgrade – but to
think strategically. This means that, from the beginning, we begin to identify
functional and organizational elements that conduct some of the same functions
and use the same (or highly similar) information. These are often termed
“Communities of Interest (COI)”. We should begin laying the foundation for
incorporating these elements after the benefits (or lack thereof) have been shown
in a small number of pilot areas.
[NOTE: Models and metrics estimated or measured at each stage should be
refined as the system becomes more refined and better models or data are
available. In addition, the specific metrics applied at each stage may be modified
based on the development methodology employed and the organizational
characteristics discussed in section 1.1.5.]
The need for this effort is motivated by changes in business (Figure 1),
Technology (Figure 2), and Standards (and terminology) that are still evolving
Business and Government are Changing
Rapidly respond to
Increased inter-organizational changes in mission,
and international collaboration business strategy,
environment in a
Tighter coupling between
customers and suppliers at
all stages of a product’s life Keep track of assets
cycle. (including information) in
structures and ensure
usability by all stakeholders.
Monitor activity to proactively
identify potential issues
before they impact Provide a consistent view of
productivity a customer / supplier across
Enabling Technology is Emerging
1988 prophecy: "The network is the computer."
An exploding array of services
Ubiquitous web are being built by using the
applications use standard software equivalent of Lego
protocols for data blocks … to "mash-up," … in
communication and new distributed applications on
representation. multiple computers. A classic
mash-up connected apartment
Software is … morphing into a service (vs. a rentals on the Craigslist Web
product in a box). It is being sold on-demand, site with Google Maps, creating
with phased delivery, and using success based- a new Web service..
Rise of transmission speeds is (having) Service registries and
a revolutionary impact on how repositories manage
computers are used…. The things that metadata describing
used to be inside your PC now (is) data and services
spread out … on a global basis," 7
Technologies and Standards are being applied to SOA
Architecture (EA) NOTE: SOA and EA (and sometimes
WS) are often used interchangeably.
Standardization e.g., BPEL4WS,
WS-BPEL, BPML, BPMN,
Execution policy, e.g., WS-TM, 3/152006: HP, IBM, Intel
SOA and Microsoft plan to
WS-SecurityPolicy, WS AtomicTransaction, develop a common set
of [Web Services]
Standards describing how to interact with a service specifications for
(e.g., WSDL), interactions between a set of services resources,
WS events and
(e.g., WS-CDL), locating services (e.g., UDDI) management
Basic data and communication standards, e.g., IP, HTTP, URL/URI,
Standards have been developed by IBM, MS, SUN, W3C,
OMG, and others. There is strong convergence at the bottom.
Less at the top (closer to the user and specific industry).
Something needs to be done. There is a lack of agreed to and validated metrics,
measurement procedures, and standards to inform system
architecture/design/acquisition. Standardization is moving slowly, metrics more
so. Since almost all systems require involvement of a system integrator, usually
with proprietary “glue” software, to build a system, a customer needs metrics to
assess alternate approaches/designs.
Additionally, appropriate models and metrics can help us evaluate compliance
with business policies (e.g., by determining the per cent of policies in a given
area, e.g., records retention, that map to a defined service or service component.
They can help us conduct more quantitative risk management, since they can
assess the extent of potential shortfalls in functionality, as in the compliance
example above, and performance.
This effort will allow more informed system design and procurement. Early
knowledge about the behavior of SOAs at scale before costly deployment should
provide huge potential savings.
As a secondary part of this study, we are working with Government agencies,
commercial users of SOA /EA technology, SOA infrastructure vendors, and
system integrators to both to refine these metrics and to determine what models /
simulations are useful and feasible (for design and implementation decisions),
which can be general enough to cover multiple technologies, environments, and
implementations, and which could be easily modified for specific technologies.
The combination of data from ongoing system development efforts and relevant
controlled experiments, related to the metrics and indicators described in this
paper, can provide the basis for a metrology of composable systems (specialized
for Service Oriented Architectures built on Web Services). They can, in addition,
provide for more objective Verification and Validation – providing quantitative
evidence if a system under development, or evolving, is likely to meet its
functional and technical requirements.
5 Document Organization
Each of the following sections outlines potential metrics and measures in the 4
stages defined above: Early stage (Section 6), Architecture/Construction (Section
7), Operations (Section 8) and, Evolution (Section 9). They are categorized,
where possible, by the FEA Measurement Areas: Processes and Activities,
Customer Results, Processes and Activities and, Technology. We emphasize
technical factors, but include some process concerns, and have an overall focus
on factors that will reduce Total Cost of Ownership.
6 Early Stage Metrics (concept development, obtaining support / coordination)
Metrics at this stage emphasize estimates/models of the system’s value in terms
6.1 Processes and Activities -- that capture the outcomes that agencies seek to
achieve and begin to identify quantitative measures of these outcomes.
Questions focus on the extent to which the system / system concept can be
mapped to, business processes. Hypothesized impacts of this mapping
include increased management support and improved co-evolution of
business and system processes. Potential metrics might reflect:
• Extent of explicit mappings between business and Service-oriented
architecture representations. Specific measures include the percent of
identified business functions and services where explicit mappings exist.
• Extent of business processes visibility and formal documentation using
BPN methodologies, allowing automated analyses of these processes
(e.g., to detect potential halting or deadlock). Specific measures include
the percent of processes and process interactions documented in a form
that allows automated analysis of logical correctness (e.g., the system
cannot reach undesirable states) and to support optimization/modification
(e.g., to adapt to changing resources or policies).
[Collection points: Initial system concept and whenever there is a
major change in business process specifications or service
[Corrective Actions: Use more formal. “analyzable” notations.]
• To what extent can the system effectively support capturing and reacting
to business events of interest by, e.g.:
o Triggering businesses processes in response to detected events
(both possibly outside the SOA)? Specific measures include the
percent of anticipated external “effector components” that the
system can “talk to” and the amount of effort involved in writing
triggering code. Is there a simple scripting language? How long
does it take to learn?
o Capturing arbitrary business events? Specific measures include the
number / percent of business events (e.g., sales last hour) in
addition to system events (e.g., percent of messages delivered)
that the monitoring process, usually part of the ESB) can capture,
the overhead involved, and the effort involved in writing this
monitoring code. Is there a simple scripting language? How long
does it take to learn?
o What effort is required to specify rules / processes for the above?
High effort (e.g., based on the absence of simple scripting
languages) is likely to increase development and maintenance
costs and decrease flexibility as business rules change.
[Collection points: Initial system concept and changes in
“events of interest”]
[Corrective actions: Buy, buy and modify, or build
infrastructure that provides needed triggers and appropriate
• Does the system have the capability to identify key aspects of processes
or activities that need to be monitored and/or improved? Potential
measures include estimates, based, wherever possible, on quantitative
models, of expected improvements in:
o (User) productivity and efficiency; Quality of information provided
with respect to appropriate accuracy, depth, breath, and timeliness
(What percent of information needed for key decisions is the
system planned to provide? How many levels and types of users is
the system designed to accommodate with tailored processes,
interaction mechanisms, and output formats?)
o Achieving financial measures related to direct and indirect total and
per unit costs of producing products and services and costs saved
or avoided. (Does the system concept include provisions for
collecting, aggregating and analyzing data relevant to a mission’s
“bottom line”? Since agency missions are often multi-dimensional,
what percent of these dimensions allow automated data collection
and analysis and reporting with respect to accomplishment of
objectives in an effective and efficient manner?)
o Providing information useful for improving management policies
and procedures, compliance with applicable requirements, risk
mitigation, knowledge management (collection and dissemination),
and continuous process/product improvement. (What percent of the
information elements that managers typically use for assessing
agency or program policies is provided by the system? Can a
manager easily define triggers to alert her to (business) policy
violations? How much effort and training are required?)
[Collection points: Initial system concept, which could define
indicators or “warning signals” for faulty processes and
whenever processes or the definition of important indicators
[Corrective action: Instrument system and data use, collect
customer feedback, analyze with respect to business
One more global evaluation of the cost/benefit of an SOA-based system (which
could fit in either Section 6.1 or 6.2) Is based on a library-science technique
called the ‘Critical Incident Technique”. In this technique a decision maker(s) is
asked to recall the last information request he (or she) had to respond to that:
• Cut across organizational or system lines (necessitating information
location and, possibly, transformation to a standard format and definition.
• Required a reasonably timely response.
• Required information analysis for summarization/aggregation, analyses of
similarities and differences (e.g., by geographical region, disease).
Concentrating on one incident will allow more specific responses than would
asking about, e.g., all information requests in the last 6 months. Averaging
multiple responses can provide a more comprehensive picture. For each request,
we collect information about:
• Date of request – assume that period between request date and
information collection date is ½ the inter-request interval – a reasonable
assumption if request frequency has a uniform distribution.
• Number organizational elements contacted and number of of separate
data stores accessed.
• Effort (time) required to collect and transform the data – including time to
negotiate access to other organizations’ data, transform the data to a
common format and definition, and collect the data. This should include
effort needed for any necessary computer programming.
• Time period from request to data/report production – latency in response.
• Estimated comprehensiveness of data obtained – comprehensiveness.
• Estimated costs to consumer of latency or lack of comprehensiveness
(e.g., extra hospital days, students not learning to read, sewerage
overflows into streams, soldiers killed or territory lost).
These figures allow us to estimate the current cost of fragmented data systems.
They can be important in justifying system modernization. Repeating the
measures in Section 8, “Operations”, can provide evidence of progress.
6.2 Customer Results capture how well an agency or specific process within an
agency is serving its customers and suggest ways of measuring these
results, e.g., using metrics/models related to:
• Service Coverage,
• Timeliness & Responsiveness with respect to the time to respond to
inquiries and requests and time to deliver products or services,
• Service Quality, both using objective measures and based on the
customer ’s perspective. This includes factors such as: Service
Accessibility, Customer Benefit, Customer satisfaction levels and tangible
impacts to customer
• In the Early Stage, these will be estimates of how well (and how much
better) the system will serve its users. Potential metrics include:
o Usability measures related to the targeted tasks (e.g., from helping
citizens evaluate health care options to supporting time critical
targeting decisions in the military). Specific measures include the
amount of time / training required for intended users to learn
targeted tasks, possibly based on experiments with prototype user
interfaces. The estimated cost of modifying interacting components
to interoperate so that the customer is presented with an integrated
picture of the situation.
o Accessibility measures. Specific measures include the time/effort
required for a customer to reach a location where they can access
the service, the length of time and amount of knowledge required to
“sign on”, the required knowledge and training, as compared with
that possessed by the intended customer(s).
o Comprehensiveness measures related to what fraction of the
targeted task is addressed by the system (e.g., using a single sign-
on). Does a user need to log on to a separate system to complete a
task (e.g., providing a user with information about health care
options, but requiring them to access (possibly multiple) other
systems to get information about available health care plan benefits
and costs)? Specific measures include the percent of a typically
integrated series of functions that the system supports in a given
[Collection points: These measures should be collected
either periodically and/or when there are significant changes
in the services being delivered or the user population]
[Corrective action: Early and extensive prototyping of
customer/system interaction, including the number of
separate systems (or people) that must be accessed to
6.3 Processes and Activities – how well (or how much better) do we estimate that
the system will capture and analyze data that contribute to or influence
outcomes that are Mission and Business Results and Customer Results?
Many measures related to this were discussed in terms of how well the
system effectively supports capturing and reacting to business events of
interest (Section 6.1) and how well it supports its users (Section 6.2).
The FEA Performance Reference Models (PRM) groups Security and Privacy in
the category of Processes and Activities, This is a subset of the general issue of
specifying and enforcing operating policies. While the Early Stage of
development cannot provide measures of actual performance in this area, it can
assess the capabilities that are required and those expected to be provided by
candidate COTS SOA infrastructures. These include consideration of whether
the system concept includes the capability for the organization responsible for a
service (or an enterprise) to specify, monitor and enforce all policies of concern
to the organization (e.g., regarding security, data formats, allowable values or
sequences of events) using simple scripting languages rather than 2- or 3-GL
Potentially useful measures include:
• The percent of policies that can be easily specified monitored and
• The percent of critical policies where significant effort/skill is needed for
specification, monitoring and enforcement.
Related measures are the expected effort and level of skill required to develop
rules for specifying and enforcing policies related to allowed, e.g.:
• Workflows, process sequences, pre- and post-conditions
• Data or commands that can be accesses/executed by specific
• Data formats, value(s), source(s)
• Data (or information) content standards (for input and output)
• Data and process privileges (security)
• Communication (or “hook-up”) privileges
• Quality of service agreements
[Collection points: Models and data should be estimated as part of infrastructure
technology evaluations, revisited during system design and implementation, and
re-collected when policies are changed.]
[Corrective action: Buy, buy and modify, or build capabilities for formally
expressing policies, including those for security and trust, and link to triggering
mechanisms to detect and prevent violations.]
During the Early Stage of development, a primary concern is whether models are
in place to predict factors such as Quality (The extent to which technology
satisfies functionality or capability requirements or best practices, and complies
with standards); Reliability & Availability (including capacity, availability to users,
and system or application failures); System cost (Total Cost of Ownership over
the system’s anticipated lifetime including costs avoided through reducing or
eliminating IT redundancies); Information (system capability to support
information & data sharing, standardization, reliability and quality, and storage),
and Behavior and Performance (e.g., response time, interoperability, user
accessibility, and improvement in other technical capabilities or characteristics).
Questions that should be answered at this stage include:
• Do we have applicable system models (e.g., of scalability, response time,
resource use, recovery from anomalies. overhead for rule monitoring and
enforcement)? Have these models been validated on similar systems? Do
they include the (often considerable) overhead required for such things as
security, run-time routing and transformation operations, etc.?
• What is the margin of error in our system models? Given this margin, are
we satisfied that the system will meet requirements using current or
planned hardware (it’s a little early for ironclad assurances)?
• What percent of services have performance requirements (Service Level
Agreements – SLAs) specified?
• Can we estimate the systems’ Total Cost of Ownership and benefits?
What is the estimated range of effort to change policy rules, services, and
service orchestration, add various types of data sources, trigger new
reports, etc.? How quickly can this be accomplished? What are the actual
and opportunity costs for the time required to modify the system?
• How does this estimated TCO compare with that of existing systems? This
might be determined using costing by analogy, since it’s too early to use
detailed parametric models or evolving Function Point Analyses
techniques at a high level of granularity.
[Collection points: At development of early conceptual models, revisited and
validated (or changed) in the Architecture/Construction stage and periodically
[Corrective action: Refine models, prototype, identify system representations that
provide more automated support for automated translations among or refinement
6.5 Evolution – All systems must evolve based on changes in requirements or
threats, policy, data availability, and equipment. The ease with which a
system can be modified needs to be considered from the early conceptual
stage. It will be influenced by the, e.g.:
• System design (e.g., modular or layered systems are easier to modify
than monolithic systems since changes can be localized) and,
• Ways in which the system is represented at various levels of refinement.
In addition, we consider evolution to include the expansion of a system to include
additional organizational units, functions, ect. This means that, from the
beginning, we begin to identify functional and organizational elements that
conduct some of the same functions and use the same (or highly similar)
information. We should begin laying the foundation for incorporating these
elements after the benefits (or lack thereof) have been shown in a small number
of pilot projects.
Many of the factors that facilitate the spread of a system (or system-of-systems
or EA) across an organization are involved in the “fit” of system components
(services) to higher levels of organizational structure and to the “two way”
potential of the SOA system incorporating (reusing) existing components and of
the larger organization reusing components developed for the SOA. The first two
elements increase an organization’s confidence, while the latter provides some
immediate benefits to organizational components not participating in pilot studies.
While most services/SOAs will be developed incrementally (e.g., to support one
function at a time), effectiveness and efficiency require that the services are
consistent with an overall organizational framework.
Potentially useful measures, whose collection/estimation begins in the Early
Stage and carries forward to design, construction, and operation, include the
percent of organizational elements or system components that have been
analyzed to determine how many, e.g.:
• Map to a more global Functional Architecture?
• Conduct the same, or extremely similar, functions (e.g., processing
electronic payments) at the lowest level of granularity?
• Use, or could use, the same specifications/design for software
components that execute these functions (at least within specified
“communities of interest”)?
• Use the same, or extremely similar, data elements, data definitions?
[Collection points: Early concept development, whenever changes
in the scope of the “enterprise” that the system serves are
[Corrective action: Improve alignment of proposed system
(architecture and data) with enterprise functional organization and
Other measures are likely to impact services’ or components’ potential for reuse,
the cost and time to evolve an enterprise-wide coordinated system, and plans for
reconciling data and functional specifications before the system is “turned on” or
adapting to the overhead required for run-time transformations. Measures
• How many (what percent of) infrastructure components and business
services will come from reuse of existing assets? This will impact: cost of
development, confidence in procedures / algorithms, feasibility of
incremental development and deployment.
• How feasible is incremental development / deployment? What is the effort
required to “wrap” and incorporate legacy systems (as a function of the
legacy systems themselves and the foundation SOA infrastructure)? This
may impact development time and cost, and the feasibility of incremental
• Are proposed new services actually reusable (what number and percent
are reusable, in how many contexts)? This can impact reduction in
development and maintenance costs, uniformity of processes across
organization. Indicators of reusability include:
o How many of the lowest level components/services are of fine
enough granularity to be incorporated in multiple higher level
o How many organizations / functions use (or plan to use) these low
o How many other organizations / functions have made (at least) a
conceptual buy-in to the concept?
• What tools are available to identify and rationalize duplicative data sets
(e.g., by analyzing the statistical distribution of data, by using data flow
analysis to identify the (original) data source).
[Collection points: Estimated during initial concept development,
refined whenever services are developed (or reused) and each time
system is expanded to include more organizational elements, data
sources, or services]
[Corrective action: Acquire and use tools for organizational data
flow analysis, component repositories, data set analysis and
Since business processes (and the SOA representing them) may change faster
than long-lived instances of a process (e.g., tracking international nuclear non-
proliferation efforts), is there a capability to run some tasks under the old process
and some under the new process and have them communicate/coordinate as
needed? How much effort is required to specify and implement this capability?
[Collection points: Estimated during initial technical evaluations, tested during
Architecture/Implementation, validated when capability is needed during
Operations and Evolution.]
[Corrective action: Select infrastructure with needed Business Process
In this stage an organization is refining requirements, conceptual designs, and
models from Early Stage activities. It is developing detailed specifications of
Services/components, and is building or buying those components needed to
flesh out the design, providing the ability to further refine and validate models
used in the Early Stage.
Many of the models and metrics developed at this and succeeding stages are
based on elaborating or refining the models and estimates/data from the
preceding stages. An important metric is: How much effort/cost is required to
update the models/data? Ideally, the tools used to produce more specific
representations of the system, or the languages/notations used to describe these
representations, could automatically update the models.
Models and metrics estimated or measured at each stage should be refined as
the system becomes more refined and better models or data are available. In
addition, the specific metrics applied at each stage may be modified based on
the development methodology employed.
7.1 Mission and Business
At this stage, we can examine the fit of the proposed system to business needs
more closely. In addition to refining models/estimates made in the Early Stage,
we can assess the extent to which the system supports consistent co-evolution of
business and technical processes. This might include metrics related to whether,
• Hierarchic designs are sufficiently linked, and refinement rules specified,
so that a change at one level of a business or service architecture
specification is reflected in the complementary architecture and at all other
levels in the architecture where the change was made? Specific
o The effort and skills needed to reflect a change in a service /
service architecture specification or a business process in the
o The effort involved in identifying what changes need to be made
and writing code to change processes or process orchestration. Is
there a simple scripting language?
How long does it take to learn?
What percent of changes processes or process orchestration
does it encompass?
• Does a machine interpretable service architecture specify important
service behavior, connection requirements and constraints?
o What percent of requirements and constraints are included?
o How much time, and what level of technical skill, is required to
specify the monitoring of business variables in addition to those
describing system performance?
• Does the system include probes, or the ability to insert probes, to monitor
services and connectors with respect to specified requirements and
o What percent of requirements and constraints are included?
o How much time, and what level of technical skill, is required to
specify the monitoring of business variables in addition to those
describing system performance?
7.2 Customer Results
Work in this area focuses on refining earlier estimates and, possibly, testing
prototype implementations developed in this stage.
7.3 Processes and Activities
These measures are intended to capture the outputs that are the direct result of
the process that an IT initiative supports. Since we are beginning to implement
the system in the Architecture/Construction stage, we may be able to measure
the impact of subsystems, if not the system as a whole. Some potential
measures were described in Section 5.3.
With respect to Policy / Security concerns, we should be able to measure, e.g.:
• What tools are available for monitoring rule conformance and what
percent of rules can be monitored using these tools?
• Is policy adherence automatically monitored and does non-adherence
trigger appropriate action (including, if required, restoration to an earlier
• Can the SOA accommodate non-symmetrical security (e.g., for data
sharing) and trust (e.g., belief in data validity) relationships.
The system cannot support business requirements unless it is sufficiently robust.
This requires, for example an adequate system-wide backup and recovery
mechanism? Potential measures include:
• What is the recovery time under various failure scenarios?
• How much information can be recovered under various failure scenarios?
• The ability to specify configuration rules, including those for sequencing
transformations and saving and restoring state information. (For what
percent of error or “out of bounds” conditions do such rules exist? What is
the effort and skill set required to write them? Are there provisions for
defining atomic transactions within and across services and conducting
appropriate “roll-backs” to handle errors?
• How much effort is required to use (or adapt) the same mechanisms used
for measuring system SLAs to monitor specified business variables –e.g.,
Other important questions relate to system performance and scalability, e.g.:
• What is impact do various configurations and different environments have
• Can we do model-based testing sensitive to the deployment
• Can the models demonstrate the system is scalable with respect to size
(number of processes, users, data stores, data throughput), required
response times and, the number and variety of systems with which it can
• For what percent of (anticipated or feasible) process flows is component
assembly and business process flow tested in addition to functional tests
on individual components/services?
• Do tests look at activity outside the application being developed and
examine interactions among all components in a total business process
(i.e., “out of the box”). What percent of “out of the box” interactions are
• How complex is the specification of data movement / migration / caching /
persistent storage? If it is incorporated as part of a service (no additional
specification required) rather than as a rule in the SOA infrastructure,
every change will require reprogramming the service.
• What services does the SOA Service Bus provide automatically (no
• How long does it take to learn any additional specification notations /
languages that are required?
• Are mechanisms for data movement and storage scalable with respect to
size (number of processes, users, data stores, data throughput), required
response times and, the number and variety of systems with which it can
• How comprehensive are (possible) rules regarding data acceptability or
• What percent of component interfaces are tested incrementally as the
components are added to the overall integration architecture?
• How are multiple data conflicts expressed / resolved (e.g., 4-5 synonyms
with slightly different data definitions)? (Indicates possible limitations on
system complexity / size.)
• How many versions of a “common” data item can be stored and
referenced to service data stores / files?
• Does the planned SOA infrastructure include negotiation tools for “self
o What features are included in negotiation process (e.g., XML
representation of data signature, process representation (formal or
informal), process/service availability, operation on unique
hardware resource, estimated time to complete task)?
o What features are missing for a component to effectively and
efficiently find and use (hook up with) a service it needs?
• Does information exchange use single messages (possibility making it
more difficult to specify multi-step transactions), exchange of messages in
a sequence, or data transfer using an industry standard (e.g., XML, HTTP,
COM, DCOM) or other open formally specified) protocols?
• Can a process/service be replaced without changing its interface?
o What is assembly cost? How much “glue code” (e.g. the ESB)
needs to be written / modified manually?
While the system is likely to undergo continuous revision based on policy and
technology changes and on customer feedback, substantial segments are now
operational. We can now refine estimates and models developed in previous
stages as well as measuring actual user satisfaction and system performance.
8.1 Mission and Business Results
Indicators in this area have been described in sections 5.1 and 6.1. We can now
collect real data with respect to these indicators. Actual outputs (e.g., claims
processed per day, at-risk patients identified, time between a warning indication
of enemy attack and response) can be measured. These are, of course,
idiosyncratic to the specific program. Another primary indicator of the system’s
success in contributing to business objectives is whether it is used. Potential
• The number and percent of potential users who are using (and funding)
services. Plotting this as a function of time can identify trends in user
acceptance (and impact on business objectives).
o Is the “consumption of services” by other organizational elements in
the enterprise tracking earlier predictions?
o Can we detect a decrease in the rate at which separate application
architectures targeted at the same functions are being developed?
What is it?
• How many (and what percent) of customers are using the system to
evaluate business rules and policies with respect to, for example,
consistency, uniform application, and impact on outcomes? How does this
compare with organizations that are not employing an EA/SOA-based
8.2 Customer Results
In addition to the usage metrics above:
• Can it’s usability by all involved personnel (from warehouse data entry to
business professionals (not systems professionals) who configure rules
and processes) be assessed?
• How long does it take an employee to become self sufficient in performing
their SOA-related task(s)?
• How much effort/skill is needed for testing new rules and processes (in the
context of the entire SOA) before incorporation?
8.3 Processes and Activities
It may be possible to measure system outputs, or trends in these outputs, at this
stage. As described in the FEA PRM, these include improvements in:
• Productivity and Efficiency
• Error rates and complaints related to products or services.
• Direct and indirect total and per unit costs of producing products and
services, and costs saved or avoided.
• Cycle Time and Timeliness
• The ability to evaluate management policies and procedures, compliance
with applicable requirements, capabilities in risk mitigation, knowledge
management, and continuous improvement.
• Security and Privacy
New technology measures that be collected during the operations stage include:
• Robustness, as described above (Section 7.4).
• Effort required to specify, monitor compliance, and define / invoke
corrective action for Level of Service contracts.
• Ability of and effort required to define checks for erroneous results (e.g.,
by defining acceptable ranges or alternative algorithms).
• Ability and effort required to specify (re)deployment of services to nodes
for load balancing.
• The percent of services that have SLAs automatically monitored and
corrective action taken as needed.
• Performance in various configurations and environments.
• Operational failure rate (e.g., MTTF).
• Stability under changes in individual services components, service
orchestration, and varying load levels.
• Are tools being used to identify conflicting (technology) policies within and
across services? These should explicitly include capabilities for defining
and monitoring SLAs, and for either adapting the system or “gracefully
degrading” the SLAs to ensure priority tasks are completed. A useful
metric might be the percent of services (at all levels of aggregation) where
this the case.
• Assuming the existence of an “Integration Competency Center (ICC)” to
specify, e.g., naming conventions, tool usage, interfaces, etc.,:
o What percent of messages are governed by ICC rules?
o What percent are automatically checked at design time? At run
It is necessary, over the system’s lifetime, to continually evaluate the factors
outlined above. It now becomes possible, in addition, to determine whether the
system representation(s) used and tools provided make it easy to modify the
system or to expand it to include more functions in the “enterprise”. Specific
measures of interest include:
• How much effort is required to specify a new monitoring capability – e.g.,
for a new business variable?
• How much effort is required to integrate a new service component (e.g.,
an additional credit check)?
• How much effort is required to mirror a change in business processes
(e.g., by changing orchestration or choreography)?
• Are there checks on policy consistency / applicability when system
changes are made?
o Are these (partially) automated?
o Is their a scripting language? How much effort is required to learn
• Can the system adapt to the use of various processes, terms, definitions
and, vocabularies across the enterprise? (Indicates possible limitations on
system complexity / size.)
o What percent of these items are expressed in machine
interpretable / analyzable representations ((e.g., BPMN, BPMEL,
ER diagrams, ontologies, rule bases)
o Are tools (e.g., services directory / repository) available to help?
How much do they reduce the effort?
o Does configuration management reflect changes to the system
(e.g., is it dynamic)? How much effort is required to identify and
register these changes?
This paper is a work in process. It has been prepared as a “strawman” to elicit
comments and suggestions from Government and Industry on:
• The validity of the indicators – are they predictive? Are some irrelevant?
• Better ways of quantifying the indicators;
• Anecdotal examples of situations where the existence of a condition has
(or has not) had a favorable (or unfavorable) impact;
• How a metrology developed around these indicators/metrics might be
most useful in Program/Project management – particularly with respect to
ongoing Verification and Validation (V&V).
• Experiments that might provide benchmark data indicating expected
ranges for different values and predict their impact on system
performance, utility, and TOC.