AMBoK ID 2047

AMPEAK Asset Management Conference 2015 AMBoK ID: 2047 Page 1
RECASTING PERFORMANCE STANDARDS TO IMPROVE SERVICE
David Zhang 1, Craig Crawley 1, Alex Nash 2
1. Sydney Water, NSW, Australia
2. Cardno, Sydney, NSW, Australia
Summary:
Sydney Water has reviewed how effectively the current System Performance Standards (SPS) in
its operating licence contribute to the Corporation’s objectives. Alternate SPSs are sought that
more accurately reflect the cost and risk balance and will drive customer service. Service that
customers most value may not be directly related to asset performance. Analysis highlight that
SPSs should closely link with value for customers. Current measures were felt to be more
indicators of system performance, not necessarily service, and could lead to inefficient
investment. In this paper, a framework has been developed to measure performance from
infrastructure investment, infrastructure performance, and service impacts to the customer
experience. Good asset management should be judged by its ability to translate investment
(inputs) to outcomes to improve system performances and customer value rather than outputs of
kilometre of renewal only. Apart from the review of the current system performance standards, it
also reviewed other water regulatory regimes such as OFWAT in UK, the Essential Services
Commission in Victoria and the Electricity & Telecommunications sectors. This provided insight
into useful ideas about performance indicators which align well with customer perceptions of
value. This paper also discusses licence limit and trend approaches in relation to driving
performance and smoothing investment efficiency. Based on the analysis, this paper
recommends to move towards an indicator basket/trend approach and to include aspects of
customer experience.
Keywords: licence, price, cost, risk, infrastructure, performance, targets, input and outcomes
1 INTRODUCTION
Sydney Water is Australia’s largest water utility, providing drinking water, wastewater services, recycled water and
some stormwater services to over 4.6 million people in the greater Sydney area. Sydney Water is a statutory state
owned corporation, with the Independent Pricing and Regulatory Tribunal (IPART) as the economic regulator.
Sydney Water’s performance targets are set by IPART in an Operating Licence every five years. The pricing of
water, wastewater and stormwater services are regulated by IPART on a four year cycle.
IPART is currently undertaking a review of the operating licence to determine if the existing licence is meeting its
objectives and whether the licence can be amended to make it more efficient. IPART considers that the primary
role of the operating licence is to ensure that Sydney Water provides an adequate level of service to its customers,
given that it is a monopoly provider of an essential service”, and is seeking to avoid regulatory duplication [1].
Sydney Water’s vision is to be the life stream of Sydney for generations to come. To achieve our vision we put the
customers at the heart of everything we do. To ensure that customers get solutions that they value, Sydney Water
would like to align the system performance standards with the delivery of customer value. With this in mind, Sydney
Water has reviewed how effectively the current system performance standards contribute to its objective of
delivering those aspects of services that matter most to customers, without adding to pressure on household bills.
Based on the review, it concludes that Sydney Water’s system performances in Operating Licence have been
performing well within the licence’s limits for the past 10 years [2].
Print Go Back Next Page

Asset Management Conference 2013 AMBoK ID: 2047 Page 2
2 BALANCE OF SERVICE, COST AND RISK
Customer value is defined as ‘worth what paid for’, a function of perceived quality of service over perceived cost (or
price). The better the perceived quality, efficiency and design of the product or service, the more likely the
customer will be prepared to pay for it, or to accept price increases.
A monopoly value proposition is different to a private company, as there is:
 less room for explicit trade-offs in service offering
 regulated prices (less ability to change price and/or structure)
 customers do not have the option to change providers.
However, monopoly service providers can still use customer value propositions to inform service priorities,
investment decisions and customer communications. For a water utility to deliver and build value it has to be seen
to be delivering on the aspects of its role that are most valued by customers – to both the majority of customers
and the most influential segments. It can also build value by fulfilling unmet needs, providing additional benefits
beyond what is expected [3].
IPART is the economic regulator to Sydney Water, which means it determines the price Sydney Water can charge
for its services. A second key function of IPART, although not an exclusive one, is to determine what level of
service Sydney Water must provide. This is done through the Operating Licence. Service levels can also be
mandated by other regulatory authorities, such as the Environment Protection Authority. IPART’s pricing
determination requires Sydney Water to operate within the defined level of services at what it determines to be an
“efficient” cost. Typically, IPART’s determination of an efficient cost is lower than Sydney Water’s estimates. Cost
reductions can be achieved by a combination of efficiencies and “service reductions” as part of a cost-service
trade-off for customers. In making these trade-offs, Sydney Water has to justify its investment programs to deliver
the agreed level of services and demonstrate the value for money to its customers. The following diagram
demonstrates the relationship between the price determination (on the y-axis) and Licence (service) standard (on
the x-axis) [4].
Where the different aspects of service can be costed, service can be combined with costs and optimised together.
Where we cannot assign a monetary value to a service, “trade-off” can be used between costs and service. Unless
we place a monetary value on a service
1
, we cannot “optimise” the service / cost “equation”. All we can do is
optimise the costs to deliver a given service. This is shown in the following diagram:
1
Placing a value on a service – for example, by conducting a willingness to pay survey
Next PageGo Back

At a given level of technology and efficiency, we can trade service for cost (the trade-off). However by either
becoming more efficient or changing technology, services can be improved without cost, or maintained at a lower
cost level.
On the cost side, we have capital and operational costs, which can be optimised by using a discount factor
2
with
consideration of other business constraints. Sydney Water has the objective of reducing capital expenditure (as
part of overall expenditure) to keep bills down for customers and to increase the efficiency of its service. In doing
so, Sydney Water has a number of different, independent axes-of action, such as:
 Optimise capex / opex using whole-of-life cost
 Minimise whole-of-life cost by doing things more efficiently
 Confirm the standard of service is “right” and supported by the customers, given the cost.
 Confirm the reliability is “right” and supported by the customers, given the cost.
 Develop & introduce new technology which can improve services, or reduce cost.
Although some of the capex savings can come from assuming higher levels of risk and exploiting headroom on
licence targets, the current reductions in capital expenditure are generally obtained through improving efficiencies,
such as:
 The water mains renewal program – more accurate financial justification, better targeting and shorter mains
lengths being replaced, resulting in the same level of risk for less expenditure
 The dry weather sewer overflow program – better targeting of CCTV (under canopy), and substitution of
some capex to opex where the whole-of-life cost / net present value is lower.
 A greater emphasis on the repeats program, which yields a better reduction in sewer chokes than the
waterways program due to the better predictive nature of a prior choke.
3 FRAMEWORK FOR EVALUATING SYSTEM PERFORMANCE STANDARDS
To be able to better understand the cost, service and risk trade-off, we need to understand how customer
experience links to the “service standard”, the asset performance and levels of investment. Capital investment is
often judged by inputs and outputs. When utilities invest capital to renew poor condition assets to improve
infrastructure performance, system reliability and reduce “service impacts”, the ultimate goal is to make sure that
customers actually experience a better level of service. The framework is outlined in the following figure by using
water main renewal as example:
2
The discount factor is a “choice”, implying that even a capex-opex cost optimisation is to some extent a “choice” reflecting the
time value of money.
Next PageGo Back

(Example)
Renew poor performance water main (decision to
make capital investment, or not)
Reducing water main breaks or leaks through
renewal poor performance water main
Reducing unplanned water interruption” or low
pressure through renewal poor performance water
main
Actual service experience depends if the customer
truly experiences water interruption when water main
fails (such as break in midnight may not have impact
on service experience)
As we can see from the diagram, part of the end “service” experienced by the customer is the result of a chain of
decisions, starting with a decision to invest capital in infrastructure (e.g. renewing water mains), through to
decisions about operating cost investment (e.g. employing more crews or call centre staff). Part of the service
experience is, of course, outside Sydney Water’s control (e.g. unforeseeable service failures due to extreme
weather events, third party damage to assets, and the customer may not read notices of planned interruptions
despite these being delivered). Sydney Water’s ability to turn investment into the service experienced by the
customer is the main function of the asset management system. A good asset management system allows a strong
link to be made between expenditure and service outcomes that customers value, while a weak or non-existent
asset management system will make it difficult for utilities to effectively channel investment into service outcomes.
The system performance standards in an Operating Licence should focus on those parts of the service that are
close to what the customer experiences, but still relatively easy to measure. Some of the licence standards are
further removed from the customer experience, and relate to the infrastructure performance (e.g. the leakage and
the response time for minor leaks which do not inconvenience customers). Perhaps even further removed from the
customer “service experience” is the “pricing process”. In this process, Sydney Water must estimate the capital and
operating investment required to meet its service standards. Between pricing and service, there is a large scope for
innovation and efficiency in expenditure. In other words, the same amount of money, used differently, can result in
very different service experiences.
Using this framework, we can imagine how the ability of utilities to translate investment into service might be
constrained. An example of this would be if the regulator set targets at the level furthest removed from the “service
experience”. This would be at the “infrastructure investment” level i.e. if system performance standards were
measured in terms of kilometres of mains renewed or upgraded. While the system performance standards might be
closely and easily linked to pricing, they would be only indirectly related to service and the utility’s ability to adapt
and seek efficient or innovative ways of delivering service would be considerably constrained. On the other hand,
price setting and performance regulation would be considerably simplified, as construction targets could be set and
checked in terms of easy-to-measure outputs.
The other extreme is for the service to be regulated close to the “customer experience” end of the spectrum, which
is only indirectly related to capital investment and pricing, and leaves a lot of autonomy (e.g. scope for innovation,
efficiency) to the utility. Getting pricing right in terms of the system performance standards then becomes harder
(for both the utility and the regulator), as the relationship with investment is less direct. This may be even further
complicated by evidence that customer satisfaction is more closely related to the “non-asset” parts of the service
experience, such as response times, their experience speaking to staff, or media reports about Sydney Water’s
activities.
In response to this trade-off – between easy-to-measure pricing inputs that do not directly reflect the service, and
outputs which are hard to price but close to what customers experience - regulators often choose indicators from
Next PageGo Back

across the spectrum. Examples include requiring utilities to report surveys of customer satisfaction or complaint
levels, and to report on overall asset and system performances, outputs of length of mains renewed to outcomes of
how customer satisfaction is improved.
4 REVIEW OF CURRENT SYSTEM PERFORMANCE STANDARDS
To improve customer’s service experience, we need to ensure that the standards adequately represent the service
that customers want so that investment decisions generate value for money. The analysis of licence standards and
customer value shows that the current set of licence standards serve a variety of functions; some are at least partly
aligned to the customer “experience”, while others are more directly aligned to investment needs or infrastructure
performance. The extent to which the indicator is within the control of Sydney Water also matters; where Sydney
Water is held to maintain standards which are beyond its control, this can drive expensive and inefficient risk
mitigation investment. It is believed that performance standards for some of the indicators can be adjusted to re-
balance the service, risk and cost paid by customers (e.g. water unplanned interruption and sewer chokes),
however changing the performance standards for other indicators would have little cost implications. The following
table demonstrates the estimated cost reductions associated with the potential changes in licence standards.
Licence standards Proposal Capex reduction Opex reduction
Low pressure Change time period to 1 hour $3m/annum Nil
Unplanned interruption Excluding 3rd
party event $15m/annum manageable increase
Repeat interruption Nil Nil Nil
Sewer choke Nil $6m/annum Manageable increase
Repeat choke Nil Nil Nil
Response time Extend to 24 hours for priority 5 job Nil 0.5m/annum
Leakage 5 year rolling average Nil 0.5m/annum
5 BENCHARKING WITH OTHER REGULATORS
In the UK, OFWAT’s “serviceability” approach considers the combined trends of a “basket” of indicators and
requires water companies to keep asset serviceability “stable” (rather than “marginal” or “deteriorating”). The
objectives for determining the rating are clearly outlined and transparent. One theoretical advantage of a “trend”
approach rather than a “limit” approach is that utilities can implement more stable expenditure programs, which can
drive procurement efficiency. Instead of expenditure fluctuating in response to performance variability for a single
asset type (which may be driven by weather patterns on sewer chokes or water main failures), expenditure is more
constant and performance is allowed to fluctuate for individual measures. The expenditure only needs to be set
such that the rolling average across the basket of performance indicators remains roughly constant.
The Essential Services Commission (ESC) is Victoria’s independent economic regulator for water, electricity, gas,
ports and rail freight industries. The key difference with the indicators used by ESC from Sydney Water’s licence is
that ESC uses “average durations” rather than the exclusive use of “number passing” indicators (e.g. % of
responses under a specified duration, number of interruptions over a specified duration). The following table shows
the electricity sector standard performance indicators. These indicators are similar to the “average duration” type
used by the ESC for the Water Sector:
Electricity Sector standard performance indicators
Indicator Definition
System Average Interruption Duration Index (SAIDI) SAIDI) = Σ Interruption Duration x customers affected / Σ total
customers
Customer Average Interruption Duration Index (CAIDI), (CAIDI) = Σ Interruption Duration x customers affected / Σ
customers interrupted
System Average Interruption Frequency Index (SAIFI), (SAIFI) = Σ Customers interrupted / Σ total customers ( =
SAIDI / CAIDI)
Momentary Average Interruption Frequency Index (MAIFI), (CAIFI) = Σ Total interruptions / Σ Number of customers
interrupted
Customer Average Interruption Frequency Index (CAIFI), (CIII) = 1 / CAIFI
Customers Interrupted per Interruption Index (CIII), (MAIFI) = Σ (device operations x customer affected) / Σ total
customers
Average Service Availability Index (ASAI) (ASAI) = [1 - Σ (interruption durations x customers affected) ] x
100
The “average duration” approach gives more insight into the failures which do occur, and keeps an incentive on the
utility to correct a fault even after the fault “counts” as an indicator failure.
Next PageGo Back

A useful statistical approach widely used in the electricity industry is the “2.5 beta method”. Because the electricity
industry methods use “average duration” type approaches, utilities can identify days when power outages are much
higher than usual in duration. These are typically days when something major has gone wrong (such as a natural
disaster or severe weather event). This approach would require Sydney Water to calculate the SAIDI score each
day
3
, take the natural log of the value obtained, and determine how many days are more than 2.5 standard
deviations from the average [5]. The advantage of removing the “major event days” from the overall indicator is that
regulators can distinguish between service failures that the utility “can control” (e.g. normal deterioration of the
asset base) compared with service failures which the utility is not expected to be able to manage (e.g. those which
result from events which are not part of the design specifications of the assets, such as severe weather events).
The telecommunications industry also offers a potentially useful indicator, which is the time to carry out specific
transactions (such as a connection to a service, or carry out a repair). This indicator clearly covers a much wider
aspect of Sydney Water’s service than restoration of supply following an interruption, and could include
administrative tasks (such as a change of address, or account name) or technical tasks (such as installation of first
connection, meter tapping). These services are known to be significant drivers of customer discontent at present,
and including them in the licence (eg replacing some existing measure) could better reflect customer priorities and
perceptions.
In any case, the introduction of a transaction-type standard should only be implemented if, following the customer
research work, it is felt that some sort of service guarantee in this area is the best way to ensure the outcomes that
customers want and are prepared to pay for.
6 CONCLUSION
Some of the current licence standards could be altered or removed from the licence with little or no impact on
customer satisfaction, but with the potential to improve the efficiency of Sydney Water’s investments and value for
money for customers. We developed a framework for considering performance measures from “inputs” to
“outcomes”: “asset investment”, “asset performance”, “asset service” and “customer experience”. Analysis showed
that although there are some strong relationships between the current licence measures, asset investment (such
as water main renewal programs and dry weather overflow programs) and customer experience, other current
licence measures (eg low pressure and water leakage targets) corresponded weakly with both asset investment
“inputs” and customer experience “outcomes”. Other sectors and regulatory regimes were able to contribute useful
ideas, notably;
 Some of the current indicators are poorly aligned to customer perceptions of value; an improvement in
performance does not improve customer satisfaction
 Some of the current indicators capture “service equity” poorly and could be changed to better measure
disparities in service between customers
 Some of the current indicators may be contradictory and not aligned to concepts such as the economic
level of leakage (eg overall leakage target; leak response times)
 Performance measures should attempt to isolate performance that management can control, and exclude
“external” factors such as weather and third party impacts.
 The inclusion of external (or uncontrollable) variance in performance indicators can drive inefficient
investment, as utilities attempt to smooth performance for indicators over which they have limited
discretion.
Analysis supports moving towards an indicator basket/trend approach for asset performance and to attempt to
include some measures closely related to customer experience. With better understanding of both the
effectiveness of the current performance targets and customer values through customer research, Sydney Water
should work with IPART to create a regulatory framework that moves away from asset performance standards and
input measures, and instead focuses on delivering outcomes that customers value, at an efficient cost. Such a
framework would need to provide safeguards to customers by defining a minimum guaranteed level of service, but
also include financial incentives for Sydney Water to provide superior performance. In addition, some existing
performance indicators could be replaced by indicators which will capture aspects of service that matter to
customers; eg a “transaction time” indicator, measuring the time it takes to get various services done. Another
indicator could be the use of trends for a basket of asset performance indicators to measure “serviceability”, rather
than specific licence limits for narrowly defined asset performance criteria. The potential changes are summarised
in the following table.
3
Sum of (interruption duration x customers affected) / total customer base.
Next PageGo Back

Summary of potential changes to licence standards
Indicator Change
Pressure Align with design standard with focus on chronic faiures
Water Supply Interruptions Remove from Licence
Repeat Water Supply Interruptions Retain in Licence
Sewer overflows Remove from Licence
Repeat Sewer overflows Retain in Licence
Response times Remove from Licence
Water conservation (WC 1) Remove from Licence
Leakage (WC 2)
Retain reference to leakage in Licence, but move target to reporting
manual to be updated with ELL
Serviceability (new standard) Consider a basket of indicators 2-3 year trends, with no targets.
Transactions (new standard) Create new standard (% transactions completed on target)
Financiability (new standard) Financial performance, avoid intergeneration equity issue
7 ACKNOWLEDGEMENTS
We gratefully acknowledge the support and assistance of Aneurin Hughes and Stephen Walker from Cardno and
Greg Kane from Sydney Water.
8 REFERENCE
1. IPART, “Review of the Operating Licence for Sydney Water Corporation Water Licensing — Issues Paper”
June 2014
2. D. Zhang, C Crawley, G. Kane, “Building level of services and customer value into decision making –
Sydney Water’s water main asset management strategy, AMPEAK 2014
3. Sydney Water’s customer value strategy – becoming a more customer-centric organisation
4. Report 1: Existing performance standard targets – Review system performance standards in Sydney
Water’s Operating Licence, May 2014
5. Report 2: Alternative approach to performance standard targets – Review system performance standards
in Sydney Water’s Operating Licence, Aug 2014
9 BIBLIOGRAPHY
David Zhang
David is a Strategist in the Service and Asset Strategy team at Sydney Water. David has 15 years of experience in
asset management, system planning and asset optimisation for water and wastewater network infrastructure with
Sydney Water, Hunter Water and Eco Water in Auckland. David has master degrees in civil engineering and asset
management.
Next PageGo Back

Craig Crawley
Craig is a Strategy Manager in the Service and Asset Strategy team at Sydney Water. Craig has over 30 years of
experience in operation, information system, asset management, and system planning for water and wastewater
network infrastructure with Sydney Water.
Alex Nash
Alex is a Chemical Engineer and senior consultant in Infrastructure, Regulation, Transactions with Cardno’s Asset
Strategies team. He has 17 years professional experience in operations and consulting, of which 13 years were
gained in the water sector in Australia, the UK, Africa and the Middle East.
Go Back

AMBoK ID 2047

Recommended

Recommended

More Related Content

Similar to AMBoK ID 2047

Similar to AMBoK ID 2047 (20)

AMBoK ID 2047