POWER SYSTEM OPERATOR PERFORMANCE: CORPORATE, OPERATIONS AND TRAINING GOALS AND KPI’S USED

677
POWER SYSTEM OPERATOR PERFORMANCE:
CORPORATE, OPERATIONS
AND TRAINING GOALS AND KPI’S USED
WORKING GROUP
C2.35
MARCH 2017

Members
N. Čukalevski, Convenor RS
C. Coluzzi IT
J. Darriba Macêdo BR
F. Villella BE
D. Kearney IE
J. Kreienkamp DE
M. Gilbert AU
M. Markovic AT
J. Muller ZA/AE
M.M. Van Ravenhorst NL
H. Sarmiento MX
R. Zacheo IT
WG C2.35
Copyright © 2017
“All rights to this Technical Brochure are retained by CIGRE. It is strictly prohibited to reproduce or provide this publication in
any form or by any means to any third party. Only CIGRE Collective Members companies are allowed to store their copy on
their internal intranet or other company network provided access is restricted to their own employees. No part of this
publication may be reproduced or utilized without permission from CIGRE”.
Disclaimer notice
“CIGRE gives no warranty or assurance about the contents of this publication, nor does it accept any responsibility, as to the
accuracy or exhaustiveness of the information. All implied warranties and conditions are excluded to the maximum extent
permitted by law”.
WG XX.XXpany network provided access is restricted to their own employees. No part of this publication may be
reproduced or utilized without permission from CIGRE”.
Disclaimer notice
“CIGRE gives no warranty or assurance about the contents of this publication, nor does it accept any responsibility, as to the
POWER SYSTEM OPERATOR
PERFORMANCE: CORPORATE, OPERATIONS
AND TRAINING GOALS AND KPI’S USED
ISBN : 978-2-85873-380-4

POWER SYSTEM OPERATOR PERFORMANCE: CORPORATE, OPERATIONS AND TRAINING GOALS AND KPI'S USED
Page 3
POWER SYSTEM OPERATOR
PERFORMANCE: CORPORATE,
OPERATIONS AND TRAINING GOALS
AND KPI'S USED
Table of Contents
Preamble ...................................................................................................................................................4
EXECUTIVE SUMMARY.........................................................................................................5
1. Introduction .......................................................................................................................................7
2. Relevant international organisations and their work in the domain of operations performance goals and
KPI’s9
2.1 Status of the ENTSO-E practice in this domain.........................................................................9
2.2 Status of the NERC/FERC practice in these domains......................................................12
2.2.1 NERC FUNCTIONAL MODEL ............................................................................................12
3. Power system operator, corporate, operations and training performance related practice-Key findings
and recommendations...........................................................................................................................21
3.1 Corporate level............................................................................................................................21
3.2 Operation level............................................................................................................................21
3.3 Training level ................................................................................................................................22
4 Questionnaire survey analysis results in more detail per level.................................................24
4.1 Power System data .....................................................................................................................24
4.2 Company Data.............................................................................................................................27
4.3 Corporate level Goals and KPI-s .............................................................................................29
4.4 Operational Goals and KPIs .....................................................................................................35
4.5 Training goals and KPI’s.............................................................................................................41
5. Case studies........................................................................................................................................49
5.1 National Grid (UK)......................................................................................................................49
5.2 ESKOM (South Africa).............................................................................................................52
5.3 ONS (Brasil)..................................................................................................................................63
6. Conclusions, Recommendations and Future work.....................................................................77
List of Abbreviations..............................................................................................................................79
Bibliography/References .....................................................................................................................82
Annex 1................................................................................................................................................... 83

Page 4
Preamble
This Technical Brochure (TB) is the result of work by the Working Group (WG) C2.35, which has recently
conducted a review of the current practices used around the world in the domain of power system operator
(TSO, ISO, RTO, VIU) performance with its goals and key performance indicators (KPI’s), but in the wider
hierarchical environment, from corporate level to operations and finally training level. The work was initiated
through the WG’s terms of reference approved in April 2013. This Technical Brochure provides details of the
previous work, of the survey, its analysis, findings and conclusions, including recommendations to utilities and
possible future work prospects in the domain.
Acknowledgements
The WG Convenor wants to thank those individuals that answered to the Questionnaire or provided other
support:
- Teresa Smit (ZA)
- Robert Brown (US)
- Jason Bareham (UK)
- Alexandre Dutoit (FR)
- Miguel de la Torre Rodriguez (ES)
- Keith Owens and Daniel Lavis (AU)
- Sam Matthews for SONI (UK)
- Haim David (IL)
- Mark Miller (AU)
Convenor also wants also to thank Mr. Goran Jakupović, from IMP Belgrade, Serbia who helped to prepare
answers, the data base and selection routines needed for analysis.

Page 5
EXECUTIVE SUMMARY
The modern electricity supply industry (ESI) challenges due to growing power system and market complexities can
be identified best during the large system disturbances and blackouts, when actual demonstrated level of
operations performance should be superior. The problem is aggravated due to the fact that the operator work
goals, and thus their performance goals, are rarely explicitly defined and quantified with suitable KPI’s and metrics.
In order to deal with large system disturbances and blackouts operators performance has to be adequately
increased and measured after the training. Feedback and quantitative measures increase the effectiveness of
training and operator capability.
It is clear that almost every performance goal and associated KPIs can be quantified and that failure to achieve an
adequate level of performance has financial consequences. Such potential underperformance should be a major
motivator for utilities to define, track, analyse and improve actual operations performance, and the basis for this
CIGRE research.
The domain explored here is related with power system operator performance, with its goals and key performance
indicators (KPI’s), in the wider hierarchical environment, where goals for the operation level should normally be
defined at the corporate (upper) level, and where operation level goals and KPI’s should be used as targets for
operator training program design and operator work evaluation (including accreditation), i.e. at the operator training
level. The term power system operator is used as generic that accommodates all different organisational/functional
forms like ISO, RTO, TSO and VIU.
Previous work regarding corporate level, operation level and training level performance goals and KPI’s is scarce.
More than two decades ago within the CIGRE SC 39 (predecessor of the SC C2) there was a WG 39.05 which
focused on BES operational performance, mainly focusing on security (operational reliability) of supply, with survey
based results (on measurement systems used and actual data on ENS and frequency of events) The results of this
work are presented in TB no.53.
Work done in recent years in CIGRE WG C2.33 that was presented in the TB 524 included vast amount of
knowledge regarding operator training, that was acquired and illustrated with several case studies from around the
world.
It is also relevant to mention previous work by the TSO Comparison Group, composed of 15-20 TSO’s, and active
from 1995 until today, with the objective to benchmark performance and exchange best practice on system
operation. About 50 performance measures were identified, but unfortunately for the interested; only about 20 of
them have been published.
Also, within the USA electricity industry, and coordinated by the FERC, with the goal to compile information and
track certain data points that are relevant to ISO and RTO performance in the areas of reliability, wholesale
electricity market performance and organizational effectiveness, 2010 Metrics Report was published, with
performance data for six main US ISO/RTO.
In accordance with the ToR, the main scope of work of this WG was focused on operations performance and its
relation with operator training goals and ultimately, operator performance quantification and measurement. The
results of the work are summarized in this document.
In the early phase of work, it was also recognized that operations performance improvement must be related to the
wider business power system performance goals that are normally defined at the corporate level. Thus the domain
of corporate level (mostly technical) goals and KPI’s was also included. This additional work involved identifying
existing utility practices in setting corporate level performance targets and their indicators and in finding (if any)
mappings between the goals/KPI’s used at different levels.

Page 6
Analysis performed on data acquired through the questionnaire based survey has revealed (i.e. concluded) that:
1. Performance goals and KPI’s exist, but normally only for the narrow domain segments, like reliability but rarely
for the whole operations and training domains.
2. The existing situation within the TSO/ISO/RTO worldwide, demonstrate many similarities in the domain of
performance goals and KPI’s used, but at the same time demonstrate differences, which are generally result of
different terminology and definitions used, but in some cases from different practices and company rules in
place.
3. The clear mapping of goals/metrics between the levels of the entire hierarchical decision making chain are non-
existent in the majority of companies. This is understandable in that currently such mappings are not an issue
in the daily work of TSO/ISO/RTO. On the other hand, the mapping issue which is not a trivial one, will become
much more important for operation performance because the general direction of electricity supply industry
evolvement in liberalized environment is moving to the situation where many of the key business components
(like cost of service, tariffs and etc.) will be recognized/paid for based on demonstrated operations
performance. In the area of operational reliability (term dominantly used in North America) performance
metrics/KPIs are very well developed and employed, while situation in Europe (ENTSO-E) in the same area
(called now security) is yet to be improved i.e. to develop suitable metrics. The work in this area is already in
progress.
4. In areas, like operational planning, training, etc. performance goals/metrics are not at the level where
operational reliability metrics development is and metrics are yet to be formalised, i.e. quantified and
internationally standardized.
5. Finally, analysis of the survey results indicates that around the world there is significant room for improvement,
through unification and standardisation of terminology, operations performance metrics and international
cooperation (through CIGRE or other international organizations).
Analysis performed on data acquired through the questionnaire based survey enabled the WG to propose several
recommendations to organisations/utilities in this domain.
1. If non-existent, companies should define and possibly quantify goals and performance indicators at all key
levels (corporate, operations, training), but at operations level at first.
2. It is recommended to companies to track power system operation performance through performance indicators
(KPI’s) on a continual basis with the help of state of the art ICT.
3. It is recommended to companies to cooperate on the issue of operations performance, to compare and
benchmark metrics and results, at the regional or even continental base.
This brochure presents results of the CIGRE WG C2.35 work in this domain, and has the following structure. In
chapter 2, relevant international organisations and their work done so far in the domain of operations performance
goals and KPI’s is reviewed, including situation in North America (NERC) and Europe (ENTSO-E).Chapter 3
presents key findings and recommendations related to power system operations performance at different levels
(corporate, operation and training). These findings and recommendations are derived from the analysis of data
acquired through a survey conducted worldwide. Chapter 4 presents the results of basic quantification and analysis
of the questionnaire answers collected from 16 participating TSO/ISOs.
Additional practical examples (as case studies) from National Grid (UK), ESKOM (SA) and ONS (Brazil) are also
presented in detail in chapter 5. The final chapter 6 contains conclusions, recommendations and suggestions for
future work.

Page 7
1. Introduction
Due to growing power system and market complexities there are increasing challenges, for modern day
TSO/ISO/RTO and DSOs, relating to power system operations performance, and consequently to operator training
and its effectiveness.
These challenges are particularly demanding during the large system disturbances and blackouts, when actual
demonstrated level of operations performance needs to be at the highest possible level. The problem is aggravated
by the fact that the operator work goals, and thus their performance goals, are rarely explicitly defined and
quantified with suitable KPI’s and metrics. In order to deal with large system disturbances and blackouts operators
performance has to be adequately enhanced through training. The effectiveness of such training in enhancing
operator capability needs to be routinely assessed through feedback and quantitative measures.
It is clear that almost every performance goal and associated KPIs can be quantified and that their failure to
achieve the required level of performance will inevitably have financial consequences.Such potential
underperformance should be a major motivator for utilities to define, track, analyse and improve its actual
operations performance, and it was also a reason for this CIGRE research.
The domain explored here is related to power system operations performance, its goals and key performance
indicators (KPI’s). In the wider hierarchical environment, such operational level goals should normally be defined at
the (upper) corporate level, and these goals and KPI’s should then be used as targets for operator training program
design and operator work evaluation (including accreditation), i.e. at the operator training level.
Previous work regarding corporate level, operation level and training level performance goals and KPI’s is scarce.
More than two decades ago within the CIGRE SC 39 (predecessor of the SC C2) there was a WG 39.05 which
focused on BES operational performance, primarily concentrating on security (operational reliability) of supply, with
results (survey based, on measurement systems used and actual data on ENS and frequency of events collected)
presented in TB no.53 [1], where other relevant KPI’s were also presented. Recently, authors of this TB (WG
C2.35) have published in Electra journal [2] the results of their initial effort in the domain of operations performance
and KPIs used.
Previous relevant work in this area was by the TSO Comparison Group, which was composed of 15-20 (numbers
were changing during the years) TSO’s, and active from 1995 until today, with some information on their site
(http://www.tso-comparison.com). Their objective was to benchmark performance and exchange best practice on
system operation. For benchmarking purposes a ‘standard TSO’ with five key System Operation processes (from
operations planning to after the fact and support processes) had been defined. Within it the Comparison Group
focus was on the development of meaningful Performance Measures (PM) for the TSO group. About 50
performance measures were identified, and about 20 of them was examples of performance measures for the five
TSO processes, published [3]. The data was collected annually from 2000 to 2013.
Also, within the USA ISO/RTO industry, as a response to the request of the U.S. Senate Committee on Homeland
Security and Governmental Affairs (in 2008) , and coordinated by the FERC, with the goal to compile information
and track certain data points that are relevant to ISO and RTO performance in the areas of reliability, wholesale
electricity market performance and organizational effectiveness, 2010 ISO/RTO Metrics Report [4] was published,
which covers the period from 2005 to 2009, with performance data from six main US ISO/RTO. Previous to this
publishing, utilities and FERC developed suitable performance metrics.
Based on CIGRE work done in recent years, as presented in the TB 524 [5], vast amounts of knowledge regarding
operator training was acquired and illustrated with several case studies from around the world. Within it, issues of
operator work performance (and related training performance) were identified but without going into deep analysis
on how operators work performance was quantified and measured.

Page 8
Thus, additional research is required, initially examining and identifying the current situation in the worldwide
domain, and leading possibly to making recommendations to utilities on what KPIs to define, acquire and monitor.
Future research may focus on identifying mappings in goals/KPIs between basic decision-making levels.
In accordance with the ToR the main scope of work of this WG, was on operations performance, its relation with
operator training goals, and consequently with operator performance quantification and measurement.
Within the operations performance and the operator performance domain/scope, the goals were to:
 Review and identify existing utility practices in operations performance targets and their indicators.
 Identify mappings between the corporate level performance goals and operations performance goals and
indicators.
 Review utility practice in operator training goals. Identify eventual indicators.
 Identify possible mappings between the operations performance goals and operator training goals.
 Develop recommendations on how to include these mappings and corresponding indicators in the training
program development framework previously developed by the WG C2.33 and presented in the TB 524.
 Identify interaction between Market Rules and Operator Performance.
In the early phase of work, it was recognized that operations performance improvement must be related to the
wider business power system performance goals that are defined at the corporate level. Therefore, the corporate
level (mostly technical) goals and KPI’s were also included. This additional work involved identifying existing utility
practices in setting corporate level performance targets and their indicators and in finding (if any) mappings
between the goals/KPI’s used at different levels.
This brochure presents results of the CIGRE WG C2.35 work in this domain, and has the following structure. In
chapter 2, relevant international organisations and their work done so far in the domain of operations performance
goals and KPIs is, including the situation in North America (NERC) and Europe (ENTSO-E). Chapter 3 presents
key findings and recommendations relating to power system operations performance at different levels (corporate,
operation and training). These findings and recommendations are derived from the analysis of data acquired
through a survey conducted worldwide. Chapter 4 presents results and analysis of the questionnaire answers
collected from 16 participating TSO/ISO. This chapter is similarly structured to the questionnaire structure (power
system data, company data, corporate level goals and KPI’s, operational level goals and KPI’s and training goals
and KPI’s).
Additional practical examples (case studies) from National Grid (UK), ESKOM (SA) and ONS (Brazil) are also
presented in detail in chapter 5. The final chapter, 6, contains conclusions, recommendations and suggestions for
future work. The references used are listed at the end of the TB.
The TB document is concluded with a list of abbreviations, while Appendix 1 contains the original questionnaire.

Page 9
2. Relevant international organisations and their work in the domain of
operations performance goals and KPI’s
2.1 Status of the ENTSO-E practice in this domain
ENTSO-E is the European Network of Transmission System Operators, which represents 42 electricity
transmission system operators (TSO’s) from 35 countries across Europe, mostly from the European Union (EU).
ENTSO-E was established and given legal mandates by the EU’s Third Legislative Package for the Internal Energy
Market in 2009, which aims at further liberalising the gas and electricity markets in the EU.
ENTSO-E has the objective of setting up the internal energy market and ensuring its optimal functioning. One of
the important issues is the integration of a high degree of Renewables in Europe’s energy system, the development
of consecutive flexibility, and a much more customer centric approach than in the past.
Within ENTSO-E there are following relevant regulations:
 ENTSO-E Network Codes (obligatory for all participants within ENTSO-E)
 RGCE Operational Handbook (obligatory for all TSOs within Regional Group Continental Europe – RGCE)
 National Grid Codes (relevant for all participants – national)
The rules are generally in this order more and more specific.
As opposed to the National Grid Codes and RGCE Operational Handbook, which have been in place for years, the
ENTSO-E Network Codes are newly developed and will introduce obligations for all participants.
In this chapter the first two international regulations will be analysed. The analysis will be completed with the
Operational Security Performance Indicators monitored within ENTSO-E, with relevant references [6-9].
2.1.1 RGCE OPERATIONAL HANDBOOK
The RGCE Operation Handbook (OH) consists of 8 Policies:
 P1 Load-Frequency Control and Performance
 P2 Scheduling and Accounting
 P3 Operational Security
 P4 Co-ordinated Operational Planning
 P5 Emergency Operations
 P6 Communication Infrastructure
 P7 Data Exchanges
 P8 Operational Training
Especially topics within P8 (operational Training) are related to the WG C2.35.
WG C2.35 analysed Policy 8 of Operation Handbook in April 2014 regarding corporate goals and KPIs. This
research shows that no defined KPI’s are inside the Operational Handbook (OH). The OH gives rules and
guidelines how to train Operators but does not specify how to measure the performance of training. Measurement
and the definition of KPI are still left to the TSOs. That is one of the main differences between current level of
development between ENTSO-E practices and NERC.

Page 10
2.1.1.1 Compliance monitoring program RGCE (self-assessment questionnaire)
A short summary and explanation of the ENTSO-E RGCE Compliance monitoring program established 2012 is
given in this section.
The self-assessment questionnaire is done on yearly by every TSO in the Regional Group Continental Europe
(RGCE). It consists of question regarding the policies of the operation Handbook (OH). Each TSO has to comply
with this and fulfil the given rules.
The Operation Handbook (OH) consists of 8 Policies. In 2014 the goal of the Compliance monitoring program was
to check the compliance of P3, P4 and P8.
Additionally, there is once in every five years per TSO an on-site visit of the compliance monitoring group, where
over two days one policy will be checked in details.
The compliance oversight report sum up the process as following:
“The objective of the Compliance Monitoring Process is to encourage TSOs’ to be compliant with the Operation
Handbook (OH) standards of the ENTSO-E Regional Group Continental Europe (RGCE). The aim of these
standards is to ensure a high level of security of supply in the RGCE synchronous area. The Compliance
Monitoring Program 2012 was put in place after introducing improvements to the methodology, which resulted from
the Compliance Monitoring Processes of 2006–2011.
The Compliance Monitoring Program 2012 was approved by the ENTSO-E Regional Group Continental Europe
Plenary on 30 November 2011. The program focussed on (TSO) self-assessment declarations and Compliance
Audits by the RGCE subgroup Compliance Monitoring and Enforcement (SG CME).”
Within the Compliance Monitoring, ENTSO-E checks the implementation of these guidelines by TSOs with help of a
questionnaire.
Table 2.1 Example: 'ENTSO-E RGCE self-assessment questionnaire 2014'
KPI are defined neither in the operational Handbook nor within the compliance monitoring.
2.1.2 ENTSO-E NETWORK CODES
ENTSO-E Network Codes are constantly developing and evolving and thus it is difficult to analyse all aspects of the
network code.
For system operations there are 3 relevant network codes:
 OS - Operational Security
 OPS - Operational Planning & Scheduling
 LFCR – Load Frequency Control & Reserves
Those were merged to the GL-SO Guideline System Operation.
Especially topics within this Guideline GL-SO (Part OS - Operational Security) are related to the WG C2.35.

Page 11
Network Code OS aims at:
a) Determining common Operational Security requirements and principles;
b) Ensuring conditions for maintaining Operational Security throughout the EU; and
c) Coordinating system operation in a common and coherent way throughout the EU.
There are no explicit corporate goals in the OS, they are mostly definitions and guidelines.
Certification of real-time Operators is required (done by the TSO), and is valid for maximum 5 years.
Training goals are defined for real time operators and system operators outside the control rooms who are carrying
out operational planning and market balancing roles.
Each TSO shall carry out training to maintain and extend the system operator employees’ skills. The detailed
contents and frequency of the training for all relevant roles shall be defined in the training program of each TSO.
The training shall include but not be limited to:
a) Relevant areas of electrical power engineering;
b) Relevant aspects of the European Internal Electricity Market;
c) Safety and security for persons, nuclear and other equipment in Transmission System operation;
d) Transmission System operation in a Normal and all other System States;
e) inter-TSO cooperation and coordination in real-time operation and in operational planning at the level of main
control centres; this part of the training shall, if not otherwise specified and agreed, be in English; and
f) Exchange and training in conjunction with DSOs and Significant Grid Users with Connection Point directly to the
Transmission System where deemed appropriate.
Each TSO shall designate at least one training coordinator who is in charge of organizing the training for the
above-mentioned group. The training coordinator ensures that each operator follows the training program.
Compliance monitoring program within ENTSO-E according to the GL-SO is to be established.
2.1.3 ENTSO-E OPERATIONAL SECURITY PERFORMANCE INDICATORS
Operational Security Performance Indicators are used for monitoring of the Operational Security in terms of
faults, incidents, disturbances and other events which influence Operational Security, as specified in the ENTSO-E
incidents classification scale developed pursuant to the Article 8(3)(a) of the Regulation (EC) 714/2009. The KPIs
are additionally specified in GL SO (Part OS)
Responsibility Areas and relevant historical information shall be approved by ACER. The annual report shall
contain the Operational Security Performance Indicators relevant to operational security:
(a) Number of tripped transmission system elements per year per TSO;
(b) Number of tripped power generation facilities per year per TSO;
(c) Energy not supplied per year due to unscheduled disconnection of demand facilities per TSO;
(d) Time duration and number of instances of being in the alert and emergency states per TSO;
(e) Time duration and number of events within which there was a lack of reserves identified per TSO;
(f) Time duration and number of voltage deviations exceeding the ranges per TSO;
(g) Number of minutes outside the standard frequency range and number of minutes outside the 50% of maximum
steady state frequency deviation per synchronous area;

Page 12
(h) Number of system-split separations or local blackout states; and
(i) Number of blackouts involving two or more TSOs.
Operational Security Ranking is used for monitoring of the Operational Security on the basis of the Operational
Security Performance Indicators, according to the ENTSO-E incidents classification scale developed pursuant to
the Article 8(3) (a) of the Regulation (EC) 714/2009;
2.2 Status of the NERC/FERC practice in these domains
2.2.1 NERC FUNCTIONAL MODEL
The North American Electric Reliability Corporation (NERC) is a not-for-profit international regulatory authority
whose mission is to assure the reliability and security of the bulk power system in North America. NERC develops
and enforces Reliability Standards; annually assesses seasonal and long‐term reliability; monitors the bulk power
system through system awareness; and educates, trains, and certifies industry personnel. NERC works with all
stakeholder segments of the electric industry, including electricity users, to develop Reliability Standards for the
reliability planning and Reliable Operation of the North American Bulk Power Systems. In the United States, from
year 2005, Reliability Standards are mandatory for all Bulk Power System owners, operators, and users. Similar
authorities are provided in Canada. NERC was certified as the ERO effective July 2006.
According to [10] the NERC Reliability Functional Model provides the framework for the development and
applicability of NERC’s Reliability Standards as follows:
The Model describes a set of Functions that are performed to ensure the reliability of the Bulk Electric System.
Each Function consists of a set of related reliability Tasks. The Model assigns each Function to a functional entity,
that is, the entity that performs the function. The Model also describes the interrelationships between that functional
entity and other functional entities (that perform other Functions). NERC’s Standards Development Teams develop
Reliability Standards that assign each reliability requirement within a standard to a functional entity. This is possible
because a given standard requirement will typically be related to a Task within a Function. A standard requirement
will be very specific, whereas a Task in the Model will be more general in nature. NERC's compliance processes
require specific organizations to register as the entities responsible for complying with standards requirements
assigned to the applicable entities. The Model is a guideline for the development of standards and their
applicability. The Model is not a Standard and does not have compliance requirements.
The NERC Reliability Functional Model that comprises three broad function categories is visualized on fig.2.1.

Page 13
Figure 2.1 NERC Functional Model Diagram, from NERC
2.2.2 Definition of terms: Reliability and ALR
For the purpose of WG study several terms are of particular importance. NERC defines [11,12] a reliable Bulk
Power System (BPS) as one that is able to meet the electricity needs of end-use customers even when unexpected
equipment failures reduce the amount of available electricity. NERC’s traditional definition of “reliability” was
ubiquitous throughout the electric utility industry, and now days in North America (starting from 2001) consist of two
fundamental concepts: adequacy and operating reliability:
Adequacy is the ability of the electric system to supply the aggregate electric power and energy requirements of
the electricity consumers at all times, taking into account scheduled and reasonably expected unscheduled
outages of system components. Adequacy is measured by comparing generation energy and capacity to forecast
net internal energy and peak demand.
Operating reliability is the ability of the electric system to withstand sudden disturbances such as electric short
circuits or unanticipated loss of system components. NERC had used the term “security” until September 2001
when security became synonymous with homeland protection in general and critical infrastructure protection in
particular. Operating reliability of bulk power system is measured by noting the response of future system scenarios
when subjected to a variety of contingencies or severe system disturbances.
Adequate Level of Reliability (ALR) is the performance state that the design, planning and operation of the Bulk
Electric System (BES) will achieve when certain reliability objectives and associated performance outcomes are
met. Term BES is used from 2005 instead of BPS (Bulk Power System).
Characteristics of a System with an Adequate Level of Reliability
The Bulk-Electric System (“System”) will achieve an adequate level of reliability [11] when it possesses following
characteristics:

Page 14
1. The System is controlled to stay within acceptable limits during normal conditions.
2. The System performs acceptably after credible Contingencies.
3. The System limits the impact and scope of instability and cascading outages when they occur.
4. The System’s Facilities are protected from unacceptable damage by operating them within Facility Ratings.
5. The System’s integrity can be restored promptly if it is lost.
6. The System has the ability to supply the aggregate electric power and energy requirements of the electricity
consumers at all times, taking into account scheduled and reasonably expected unscheduled outages of system
components.
The System exhibits an adequate level of reliability when it possesses these six characteristics. Some of the terms
such as “acceptable limits” and “acceptable performance” require specificity in order to be applied.
The definition of adequate level of reliability is broad enough to apply to all possible NERC standards, and
therefore it is not based on specific metrics. However, NERC have developed metrics at the System level that track
performance of these characteristics. These System performance metrics are different from metrics in a
standard which are used to determine compliance. System performance metrics will provide feedback for
improving the Reliability Standards. They will help identify reliability gaps and point to existing standards that need
to be modified or new standards that need to be developed.
2.2.3 NERC RELIABILITY STANDARDS
2.2.3.1 Definition of a Reliability Standard
A Reliability Standard [13] includes a set of Requirements that define specific obligations of owners, operators, and
users of the North American Bulk Power Systems. The Requirements shall be material to reliability and
measurable. A Reliability Standard is defined as follows:
“Reliability Standard” means a requirement to provide for Reliable Operation of the Bulk Power System, including
without limiting the foregoing, requirements for the operation of existing Bulk Power System Facilities, including
cyber security protection, and including the design of planned additions or modifications to such Facilities to the
extent necessary for Reliable Operation of the Bulk Power System, but the term does not include any requirement
to enlarge Bulk Power System Facilities or to construct new transmission capacity or generation capacity
2.2.3.2 Reliability Principles
NERC Reliability Standards are based on certain reliability principles [14] that define the foundation of reliability for
North American Bulk Power Systems. Each Reliability Standard shall enable or support one or more of the
reliability principles, thereby ensuring that each Reliability Standard serves a purpose in support of reliability of the
North American Bulk Power Systems. Each Reliability Standard shall also be consistent with all of the reliability
principles, thereby ensuring that no Reliability Standard undermines reliability through an unintended consequence.
The principles are:
Interconnected bulk power systems shall be planned and operated in a coordinated manner to perform reliably
under normal and abnormal conditions as defined in the NERC Standards.
The frequency and voltage of interconnected bulk power systems shall be controlled within defined limits through
the balancing of real and reactive power supply and demand.
Information necessary for the planning and operation of interconnected bulk power systems shall be made
available to those entities responsible for planning and operating the systems reliably.
Plans for emergency operation and system restoration of interconnected bulk power systems shall be developed,
coordinated, maintained, and implemented.
Facilities for communication monitoring and control shall be provided, used, and maintained for the reliability of
interconnected bulk power systems.
Personnel responsible for planning and operating interconnected bulk power systems shall be trained, qualified,
and have the responsibility and authority to implement actions.
The reliability of the interconnected bulk power systems shall be assessed, monitored, and maintained on a wide
area basis.

Page 15
Bulk power systems shall be protected from malicious physical or cyber-attacks.
2.2.3.3 Market Principles
Recognizing that Bulk Power System reliability and electricity markets are inseparable and mutually
interdependent, all Reliability Standards shall be consistent with the market interface principles. Consideration of
the market interface principles is intended to ensure that Reliability Standards are written such that they achieve
their reliability objective without causing undue restrictions or adverse impacts on competitive electricity markets.
2.2.3.4 Types of Reliability Requirements
Generally, each Requirement of a Reliability Standard identifies a Functional Entity (see sec.2.2.1), and the
conditions to achieve a specific reliability objective. Although Reliability Standards all follow this format, several
types of Requirements may exist, each with a different approach to measurement.
Performance-based Requirements define a specific reliability objective or outcome achieved by one or more
entities that has a direct, observable effect on the reliability of the Bulk Power System, i.e. an effect that can be
measured using power system data or trends. In its simplest form, a performance-based requirement has four
components: who, under what conditions (if any), shall perform what action, to achieve what particular result or
outcome.
Risk-based Requirements define actions by one or more entities that reduce a stated risk to the reliability of the
Bulk Power System and can be measured by evaluating a particular product or outcome resulting from the required
actions. A risk-based reliability requirement should be framed as: who, under what conditions (if any), shall perform
what action, to achieve what particular result or outcome that reduces a stated risk to the reliability of the Bulk
Power System.
Capability-based Requirements define capabilities needed by one or more entities to perform reliability functions
and can be measured by demonstrating that the capability exists as required. A capability-based reliability
requirement should be framed as: who, under what conditions (if any), shall have what capability, to achieve what
particular result or outcome to perform an action to achieve a result or outcome or to reduce a risk to the reliability
of the Bulk Power System.
The body of reliability Requirements collectively provides a defence-in-depth strategy supporting reliability of the
Bulk Power System.
2.2.3.5 Elements of a Reliability Standard
A Reliability Standard includes several components designed to work collectively to identify what entities must do
to meet their reliability-related obligations as an owner, operator or user of the Bulk Power System. The
components of a Reliability Standard [15] may include the following attributes: Title, Number, Purpose,
Applicability, Effective Dates, Requirement, Compliance Elements, Version History, Variance, Compliance
Enforcement Authority, Application guidelines and Procedures. The only mandatory and enforceable components
of a Reliability Standard are the: (1) applicability, (2) Requirements, and the (3) effective dates. The additional
components are included in the Reliability Standard for informational purposes, to establish the relevant scope and
technical paradigm.
2.2.3.6 The Reliability Standard Groups
The individual reliability standards are organized in 14 groups, focused on basic BES operational issues:
 Resource and Demand Balancing (BAL)
 Communications (COM)
 Critical Infrastructure Protection (CIP)
 Emergency Preparedness and Operations (EOP)
 Facilities Design, Connections, and Maintenance (FAC)
 Interchange Scheduling and Coordination (INT)
 Interconnection Reliability Operations and Coordination (IRO)
 Modelling, Data, and Analysis (MOD )
 Nuclear (NUC)
 Personnel Performance, Training, and Qualifications (PER )
 Protection and Control (PRC)

Page 16
 Transmission Operations (TOP)
 Transmission Planning (TPL)
 Voltage and Reactive (VAR)
Their detailed description can be found in [13]. In total, there are 112 individual reliability standards. They are
available at [http://www.nerc.com/pa/Stand/Reliability%20Standards%20Complete%20Set/RSCompleteSet.pdf]
within the voluminous (2456 page, as of 26 January, 2017) document.
Also, for the purpose of easier comparison, reliability standards have been grouped in the following (8) standard
objective areas:
 Reliability Planning and Operating Performance
 Frequency and Voltage Performance
 Reliability Information
 Emergency Preparation
 Communications and Control
 Personnel
 Wide-area View
 Security
2.2.3.7 Reliability Assessments
Currently [20] within NERC following annual reliability assessments are performed:
Long-Term Assessment (annual ten-year reliability assessments, provide a technical platform for important policy
discussions on technical challenges facing the interconnected North American Bulk-Power System)
Seasonal Assessments (Identify and report on the electric industry’s preparations to manage potential seasonal
issues for both the winter and the summer)
Special and Scenario Assessments (Deeper focus on issues/emerging risks identified in Long-Term Reliability
Assessment)
Long Term Reliability Assessment (LTRA) and Seasonal reliability assessments (SRA) provide insights on future
BES challenges, potential gaps, and solution sets to help mitigate these challenges, with typical scope (key
findings type) presented below.
Figure 2.2 Insights on future BES challenges from LTRA and SRA

Page 17
These reports are based on data and information submitted by each of the eight Regional Entities submitted twice
a year and periodically updated.
2.2.4 Compliance Monitoring
The compliance monitoring process has a goal to monitor, review, and ensure compliance of responsible entities
with reliability standards. The primary purpose of compliance analyses (and reporting) is to provide information on
compliance including reasons for violations and identification of process enhancements and lessons learned to
assist Registered Entities in improving compliance and thus enhancing reliability.
Time horizons are used as a factor in determining the size of a sanction in case of non-compliance. If an entity
violates a requirement and there is no time to mitigate the violation because the requirement takes place in real-
time, then the sanction associated with the violation is higher than it would be for violation of a requirement that
could be mitigated over a longer period of time.
When establishing a time horizon [16] for each requirement, the following criteria should be used:
Long-term Planning — a planning horizon of one year or longer
Operations Planning — operating and resource plans from day-ahead up to and including seasonal.
Same-day Operations — routine actions required within the timeframe of a day, but not real-time.
Real-time Operations — actions required within one hour or less to preserve the reliability of the bulk electric
system.
Operations Assessment — follow-up evaluations and reporting of real time operations
Key Compliance Monitoring Index
Key Compliance Monitoring Indicator is a historical measure that is used in NERC for trending. The KCMI does not
address real-time or forward-looking compliance performance. It measures compliance improvement based on a
set of 26 key reliability standards requirements. The index increases if the compliance improvement achieved over
a trending period. The violations included in KCMI all have high violation risk factors (VRFs) and actual or potential
serious reliability impacts.
2.2.5. BES SYSTEM RELIABILITY METRICS PERFORMANCE
The intent of the set of NERC reliability standards is to deliver an Adequate Level of Reliability. The latest set of
characteristics associated with an Adequate Level of Reliability is posted on the Reliability Standards Resources
Web Page [17].
For establishing Reliability Performance Benchmarks proposed is metrics for determining an adequate level of
reliability. The reliability performance data used for this metric could include, but is not limited to, data related to
system limits and actual system conditions. The Adequate Level of Reliability Indices (ALRI) is designed to help
measure reliability based on bulk power system characteristics, operations data and planning assessments.
2.2.5.1 Reliability Performance indices
The NERC Operating and Planning Committees approved eighteen metrics (KPI’s) that address the characteristics
of an adequate level of reliability (ALR). All current (18) reliability indicators (ALR’s) and their allocation within
relevant performance category (5 or 6 characteristics of the ALR) are given in a table form in the sequel.

Page 18
Table 2.2 NERC Reliability performance indices
Their detailed explanation can be found in [17].
2.2.5.2 Relation of ALR characteristics with the standard objectives
The way ALR indices are related with specific NERC reliability standard group is indicated on next table, extracted
from the recent NERC document [18].
Table 2.3 Adequate Level of Reliability Characteristics
It can be noticed that not all reliability standard groups are covered with related ALRI.
These metrics exist within a reliability framework and overall, the performance metrics being considered address
the fundamental characteristics of an ALR. Each of the performance categories being measured by the metrics

Page 19
should be considered in aggregate when making an assessment of the reliability of the BES with no single metric
indicating exceptional or poor performance of the power system.
In developing eighteen metrics to measure adequate level of reliability, any such measurement of reliability must
include consideration of the risks present within the bulk power system to appropriately prioritize and manage these
system risks. This approach not only can be used to measure risk reduction over time. Also, a uniform event
analysis process can be applied to identify significant events for further detailed review.
The NERC Operating Committee and Planning Committee endorsed the concepts and framework of the risk‐based
approach in their September 2010 meetings and further supported the event severity risk index (SRI) calculation in
March 2011 [19].
2.2.5.3 Event Severity Risk Index (SRI)
Risk assessment is an essential tool for achieving the alignment between organizations, people and technology.
This will assist in quantifying inherent risks, identifying where potential high risks exist, and evaluating where the
most significant risk reduction can be achieved.
The SRI is a daily, blended metric where transmission loss, generation loss, and load loss events are aggregated
into a single value that represents risk to the system. Each element (transmission, generation, and load loss) is
weighted by the inventory for that element to rate significant events appropriately. On a yearly basis, these daily
performance measurements are sorted in descending order to evaluate the year‐on‐year performance of the
system.
SRI event = (RPL)*wL*(MWL) + wT*(NT) + wG*(NG)
Where:
SRI event = severity risk index for specified event (assumed to span one day),
wL = 60%, weighting of load loss,
MWL = normalized MW of Load Loss in percent,
wT = 30%, weighting of transmission lines lost,
NT = normalized number of transmission lines lost in percent,
wG = 10%, weighting of generators lost,
NG = normalized number of generators lost in percent,
RPL = load Restoration Promptness Level:
RPL = 1/3, if restoration < 4 hours,
RPL = 2/2, if 4 <= restoration < 12 hours,
RPL = 3/3, if restoration >=12 hours
Performance Analysis-- Its purpose is to objectively provide an integrated view of reliability performance.
As previously explained it is based on SRI (Event Severity Risk Index) and other ALR (Adequate Level of
Reliability) indicators monitoring and trending, used for further analysis and identification of areas for improvement.
Comparison of NERC and ENTSO-E relevant practices
The NERC website and the documents are very structured. The NERC appears to have built up their framework
from the top-down. All levels fit together and the different documents refer to each other. On the highest level, they
have a Functional Model defined. ENTSO-E is still developing from policies to Guideline and Network Code. There
is currently luck of structure for the ENTSO-E documents. A lot of operational documents, reports and metrics at
ENTSO-E are similar to NERC. In NERC reports lessons learned are defined, and actions (sometimes training and
development of employees) recommended in order to achieve better results (metrics) on reliability of the system.
Two parts of these reports, from the human side point of view, are important to mention in this document, Human
factors and Certification and Examination.
Human Factors
Human factors are important differences between ENTSO-E and NERC. ENTSO-E is starting to mention the
importance of human factors in operating and working on a high voltage grid whereas NERC is more advanced

Page 20
organizing Human Factors conferences every year. ENTSO-E has only a compliance check; there are no metrics,
or reporting of actions taken on these soft skills. In contrast, NERC annual reports consist of metrics of human
errors. In these reports actions are also recommended if metrics on human error are too high. These actions can
consist of training and/or adaption of software or equipment.
Certification and examination
Certification and examination in ENTSO-E area is the responsibility of the TSOs. There is only a check for
compliance with the requirements for training and certification. Most of the TSOs do the recertification of operators
based only on attendance and not based on an exam or assessment. At NERC the certification and examination is
centralized. The certification is based on education (training course attendance) and examination. The
development of training plan, examination and certification is done with input from reports such as annual reports,
near miss reports and lessons learned.

Page 21
3. Power system operator, corporate, operations and training
performance related practice-Key findings and recommendations
3.1 Corporate level
Most organisations define their corporate goals through corporate KPIs. How these goals are mapped to corporate
KPIs differs in each organisation. The majority of corporate KPIs are set annually and for the most part are fixed
internally. The main exception to this is the goal reflecting an integrated energy market which is equally divided
between being set by internal and external factor (e.g. regulator). The KPIs in the majority of organisations are
monitored by either a Business Intelligence/Data Analysis or a Traffic Light system.
In general, there is large range of diversity in the different organisations in the mapping of goals to corporate KPIs.
Some of the most diverse responses were found in the corporate goals relating to ‘To ensure a safe, reliable and
uninterrupted supply of electricity’, ‘Drive growth/further strengthen of leading position’ and ‘People Engagement’.
The corporate KPI relating to ‘operational excellence’ is divided equally between ‘Interruptions, Frequency Control
and Voltage control’. The Corporate Goal ‘Innovation Simulation’ is interestingly mapped to ‘network capacity’.
Some operational KPI’s are mapped exactly to the corporate KPIs e.g. Incident Frequency Rate /Lost Time
accidents/Lost time injury rate. In general, however operational KPIs are more quantitative than their corporate
KPI’s. An example is the Operational KPI ‘Demand Forecast error’ which is linked to the Corporate KPI
‘Performance Monitoring’ and Operational KPI ‘Outage Realisation Costs’ is linked to the Corporate KPI
‘Transmission Capability’.
The mapping of Corporate KPIs from Corporate Goals varies largely in organisations and this may be due to the
qualitative aspects of these goals and KPIs. If organisations wish to standardise their corporate goals/KPIs
adopting quantitative corporate Goals and KPIs would allow this with greater ease. On the other hand, having
qualitative goals and KPIs may be a benefit to the company as it provides them with the scope to change their
corporate KPIs to suit the business needs of the organisation in that particular period.
3.2 Operation level
Although we have found that all the answers in the survey indicated that the TSO/ISO surveyed have their
operational objectives defined, one third of this group have no KPI measurement for verification and evaluation of
compliance with the goals.
One possibility may be the fact that these companies define more abstract goals that require less measurement to
achieve these goals.
We cannot exclude the possibility that representatives of TSO/ISO respondents are not involved in establishing the
KPI assessing these operational objectives and, therefore, the answer is masked by lack of specific knowledge by
some of respondents. It is possible that in some companies the evaluation of KPIs is being carried out in non-
operational areas. This possibility would require extending the depth of the questionnaire, but even without this, it
does not seem to be a good practice to decouple the evaluation group of the operational objectives from the
executor group of the operation itself. In this situation, there is the risk that the assessment would fail and not
generate immediate feedback and gains in achieving the objectives.
Observations from TSOs/ISOs that use KPIs to verify operational objectives, show, as in the first three answers to
question16 (see Annex1), that traditional indicators with easy and objective evaluation appear more frequently
among the companies surveyed. This result seems to imply that objectives are met with regard to maintaining
network reliability and capacity of supply. The evaluation of the results is verified by quantitative indicators such as
frequency variation and energy not supplied.
With respect to the goal on operational practices, a variation of the traditional KPIs can be evidenced by the higher
frequency of the KPI operation capacity planning, probably in anticipation of the difficulties that could be observed
in real time. The evolution of computer resources that enable better forecasts and planning improvements is
certainly contributing to this increased emphasis on the evaluation of these items by linking them to operational
goals.

Page 22
It is noteworthy that other indicators linked to the goal, operational practices, are still significant, including the
evaluation of the quality of monitoring and measurements received and the management of the electrical system
equipment maintenance, keeping the relationship between the indicators and the defined goals.
Examination of answer 3 of question 16 (see Annex 1), where the link to the KPI to assess the operational and
environmental safety issues, the answers received show the greatest simplicity in measuring the actions that
guarantee or compromise the security compared to environmental issues.
With regard to economical operations goals and KPIs observed in the development of the operation tasks, the
range of responses indicate that they remain important but are addressed in very different ways. It is likely that
these approaches are linked to the different realities of each surveyed TSO/ISO. The various operating methods
and market rules require different operational goals and this difference creates a wider range of indicators that best
fit the measurement needs in each situation.
A significant number of surveyed operators keep the reliability in operation despite its cost in real-time. It is
noteworthy that among the TSO/ISO surveyed are those whose real-time operating philosophy practically excludes
commercial issues when achieving the goal of maintaining supply reliability.
Analyses of the indicators demonstrate a strong link between the operational goals and customer satisfaction.
Evidence of frequent measurement of customer perception exists and despite a small range of indicators identified,
the concern that motivates them all is consistent and leaves no doubt about the focus of this area in establishing
operational goals.
However, there are minimal indicators measuring employee satisfaction and development. This fact shows that
many companies are not defining operational objectives linked to these themes and thus avoiding the need for
measurement.
The option "other indicators" in the survey specifies indicators relating to the topics already mentioned, and
excluding the possibility of mistakes in research, may signify that approaches to the same indicator may be
different due to the variances in TSOs/ISOs.
Important indicators relating to the training of operations staff and the application of budget in these activities
appear in this part of the questionnaire. Knowing in advance that the structures designed for training are different in
each TSO / ISO, the presence of these indicators, at least, denotes that Operational Goals do not exclude the
concern for the training teams.
3.3 Training level
Training is performed in all companies that answered the Questionnaire but of course in different forms and
different KPI's. Also, use of the related KPI's is evident in all of the answers. The organization and coordination of
training is done internally in all companies. There are different structures in companies, in some there are
employees that work part time and in others full time on organizing, performing or evaluating training.
Evaluation of the external companies that perform training was not covered in this questionnaire. The questions 21
to 33 have not covered the content of training in detail but the answers to Questions 23 and 24 give a more
detailed look into training methods.
Measurement of training outcome is done with or without a test which differentiates the training KPIs.
All training takes into account the structure of the system they operate, especially regarding neighbouring TSOs
and rules within the operated area.
The question about training goals reveals that the traditional aspects of operation training relating to normal
operation and operation under emergency conditions are still performed.
However, when encouraged to present other training goals, surveyed companies show training focused on
communication with neighbouring operators, including knowledge of English as the common language. This
certainly confirms the growing concern with interconnected operator actions.
Some responses received, lead to the need for analysis in the importance of training in soft skills, in the future.

Page 23
Regarding the KPIs used for training management, the conclusion is that the process of achieving operator
certification is not the primary focus of the majority of the companies, although this does not mean that they give
low importance to this certification.
The focus of management by KPIs remains more frequent in the qualitative evaluation of the training activities.
The existence of a formal training coordinator, an obligation imposed on companies participating in the ENTSO-e,
can serve as support for the establishment of a formal structure for power system operator training in other
companies, in particular, where there is no such structure.
Future research may compare advantages and disadvantages of the existence of an internal training group, who
appeared as a frequent answer in the surveyed companies, compared to the structure where there is only the
formal figure of the training coordinator.
The answers on the measurement of training outcome focus on certification, as well as the methods of training
used and demonstrate the common feature that all possibilities can be effectively used, both in the different type of
evaluation tests and in different methods of training application.
Evaluation by external companies and e-learning does not have significant use among those surveyed.
These answers are consistent with previous operational indicators and power system operators training indicators,
with a strong link to system reliability and operation under normal and emergency conditions. It seems logical that
any training method or measurement available that focuses on operator’s performance should be used.
For this reason and, of course, due to the evolution of the technical tools, use of DTS/OTS has significantly
increased in companies.
The issue of joint training with all involved agents (ISO/TSO/DSO/PP) is directly related to the growing importance
of interconnected operation between different countries or within the same country. The variation between the
answers received shows the existence of a relationship between the different system structures.
Finally, the question on the translation of training goals into individual training plans, which was to be answered in
the form of free comments, indicated great content dispersion. Only the traditional evaluation of skill weaknesses
generated an update or new training plans. A deeper study on best practices for structuring training plans through
specifically targeted questions (focused training needs analysis) could be the basis for future research.

Page 24
4 Questionnaire survey analysis results in more detail per level
The purpose of this section is to summarize some of the relevant data that characterizes the electric power system
of the companies surveyed, as well as data relevant to the purpose of this TB.
This chapter includes the results from the questionnaire sent out by the work group to international Electrical
Utilities and the respective responses from 16 Grid Operators on corporate, operational and training goals and
KPIs. The chapter then shortly describes the statistics obtained and findings and conclusions derived from the
analysis of the data.
Most of the graphs shown below are built showing the number of answers for each question from each company
surveyed. For example, in Fig. 4.1, five companies answered having voltage levels in the 200-399 kV range.
Regarding the methodology, the questions presented in the questionnaire were answered by respondents using a
drop down menu with predetermined answers identified during information gathering session performed by the
workgroup previously. In each there was an option to select “Other” with a space to describe when different from
the predefined answers.
4.1 Power System data
a) Voltage level
Of the companies surveyed, more than half (56%) use voltage levels of 400 kV or more in their transmission
systems, as illustrated in the graph of Fig. 4.1.
Question 1 : Nominal voltage levels
Figure 4.1 Transmission voltages used
b) Peak load
Fig 4.2 shows how the peak load is distributed among the companies surveyed (number of companies with a
certain peak load range); there is a broad range of values, from 1,733 MW to 102,098 MW, with a median of
34,362 MW.

Page 25
Question 2: Peak load
Figure 4.2 Peak load values distributed among the companies surveyed
Regarding the peak load season, this is split evenly: 50% of the companies surveyed have a winter peak; the other
fifty has a summer peak.
c) Power System extension
Another important indicator of the company size is the total length (in km) of its electrical lines. As can be observed
from Fig. 4.3, a third of the companies surveyed have more than 50,000 km of power lines: Five companies
reported having 50,000 km or more of electrical lines).
Question 3: Power System extension
Figure 4.3 Kilometres of power lines distributed among the companies surveyed
d) Installed generation capacity
Again, the companies surveyed comprise a broad range of values in installed capacity, from a minimum of 3,348 to
a maximum of 182,000 MW. Considering the numbers reported, this range has a median of 44,281 MW and an
average of 64,887 MW. Fig. 4.4 shows the number of companies that reported a certain range of values.

Page 26
Question 4: Installed Generation Capacity
Figure 4.4 Number of companies reporting a range of values in installed generation capacity
To have an idea of the generation mix when taking into account the total number of companies surveyed, Fig. 4.5
shows the percentage of each generating technology, considering the sum of the installed generation capacity.
Question 4: Capacity (MW) per generation technology used in Control Area
Figure 4.5 Percentage of generation technology considering the total amount of installed capacity
It´s interesting to note that renewable energy sources (RES) comprise 34% of the installed capacity. Also
interesting to observe is that coal is still the fossil fuel with the greatest share (32.3%).
In summary, survey results indicate that the sample of companies that answered the survey shows a wide variety
of size, peak load, extension and generation mix. Hence, the sample is considered adequate for the analysis
shown in the rest of this chapter.

Page 27
4.2 Company Data
a) Type of Company
As per the type of company that answered the questionnaire, Fig. 4.6 shows three types: Independent System
Operator (ISO – does not own network assets), Transmission System Operator (TSO – owns a greater portion of
the high voltage interconnected network), and Transmission System and Market Operator (TSMO).
Question 5: Type of company
Figure 4.6 Company type
In short, all companies surveyed operate at liberalized electricity market, with the TSO being the dominant figure.
b) Number of employees
The number of employees is also indicative of the quality of the sample. Fig 4.7 displays the number of employees
within a range of values. Approximately a third of the companies surveyed report more than 2000 employees. On
the other hand, roughly a third of them state they have between 0-600 employees. Again, the spread in number of
employees is indicative of the diversity and adequacy of the sample.
Question 6: Number of employees
Figure 4.7 Number of employees reported within a range of values
c) Number of control rooms
Relevant to the objectives of this TB is the number of operators, number of control rooms and operators per type of
responsibility. These statistics are discussed in this section and the next one.

Page 28
The bar graph in Fig. 4.8 shows the distribution of the number of control rooms among the companies surveyed.
Question 8 : No. Of Control Rooms
Figure 4.8 Number of control rooms
It is noted that 53% of the companies report having between 1 and 4 control rooms; while the remaining 47% have
between 5 and 8 control rooms. That is, number of control rooms is evenly distributed among the sample.
d) Total number of operators and operators per type of responsibility
With reference to the total number of operators; as illustrated in Fig. 4.9, 50% of the companies report having
between 0-50 operators, while only two of them have between 200-300 operators.
Question 7 : Number of Operators
Figure 4.9 Numbers of operators in the given range-distribution of answers
The questionnaire also asked companies for the number of operators per area of responsibility; identifying four of
them:
 System responsible
 Grid responsible
 Market responsible
 Operational Planning responsible
3
2
1
2
3
2
1
1
0 0,5 1 1,5 2 2,5 3 3,5
1 control room
2 control rooms
3 control rooms
4 control rooms
5 control rooms
6 control rooms
7 control rooms
8 control rooms

Page 29
Regarding the first two, more than 50% of the companies stated that between 1 and 3 operators are responsible for
these areas.
With reference to the last two (market and operational planning responsible), half of the companies reported having
none or one operator for these responsibilities.
4.3 Corporate level Goals and KPI-s
Questionnaire Analysis (Questions10-14, see Annex 1)
The vast majority of the organization (81% of the answers) have corporate goals defined though corporate KPIs.
Principal corporate and operational KPIs used that are related to the different corporate goals, will be presented in
the sequel. The Corporate goals divided into six categories as listed below with questions on the mapping of
corporate KPIs to Corporate Goals and then Operational KPIs from Corporate KPIs. The respondent can choose
up to 5 KPIs for each goal/KPI. The corporate goals are:
 To ensure a safe, reliable and uninterrupted supply of electricity
 Drive growth/further strengthening of leading position
 Integrated energy market
 Deliver Operational Excellence and Quality Efficient Operations
 Engage people
 Stimulate Innovation
Question 11: What KPIs are linked to the goal ‘To ensure a safe, reliable and uninterrupted supply of electricity’?
The KPI’s related with the goal: To ensure a safe, reliable and uninterrupted supply of electricity is shown in the
Fig.4.10 below. Most of the corporate KPIs linked to this goal are fixed externally (e.g. Regulator, government
body) and can be defined quantitatively such as energy not supplied and reliability.
Figure 4.10 Corporate KPIs related to the corporate goal: To ensure a safe, reliable and uninterrupted supply of electricity
Question 11: What corporate KPIs are linked to the goal Drive growth/further strengthening of leading position?

Page 30
With regards to corporate KPIs related to the growth and the future company perspectives, they are mostly related
to performance monitoring together with project delivery and growth of the assets, 80% of such KPIs are fixed
internally. Figure 4.11 demonstrates this.
Figure 4.11 Corporate KPIs related to the corporate goal: Growth
Question 11: What corporate KPIs are related to the goal ‘Integrated Energy Market’
Equally internally and externally defined (i.e. by for example a national regulator) are the KPIs related to the
development of an integrated energy market and to the delivery of operational excellence, with the most frequent
being the one related directly to the quality of supply to the final client. Figure 4.12 displays these results.
Figure 4.12 Corporate KPIs related to the corporate goal: Integrated energy market
Question 11: What corporate KPIs are related to the goal ‘Deliver Operational Excellence and Quality Efficient
Operations?
Figure 4.13 shows that the goal ‘Deliver Operational Excellence and Quality Efficient Operations’ are in the majority
of cases defined by voltage control, frequency control and supply interruptions.
Figure 4.13 Corporate KPIs related to the corporate goal: Delivery of operational excellence

Page 31
Question 11: What corporate KPIs are related to the goal ‘People Engagement’?
The customer satisfaction score is the most frequent corporate KPI for people engagement, together with the staff
training. Such KPIs are in most of the cases (86%) defined internally.
Figure 4.14 Corporate KPIs related to the corporate goal: People engagement
Question 11: What corporate KPI is related to the goal ‘Innovation Stimulation’?
The network capacity is the most common corporate KPI related to the innovation stimulation, this is strongly
related to the results for the creation of an integrated energy market.
Figure 4.15 Corporate KPIs related to the corporate goal: Innovation stimulation
Question 11: What other corporate KPIs are used?
Other corporate KPI appearing rarely in the questionnaire are shown in table 4.1 below.
Table 4.1 Other corporate KPI
Accident rate and severity
Budget
Customer queries
Deliver planning studies
Demand forecast errors/ Demand Forecast Assessment
Energy Not Supplied per year/ System Minutes Lost < 1/ System Minutes Lost >1
Human error self-declaration
IT/OT availability
Maintenance Completed/% of planed inspections, revisions, functional tests and repairs fulfilled
N-1 violation
Number of Environmental contraventions of legislation
Resources & Capabilities to Deliver
Water System Performance
1
1
1
8
0 1 2 3 4 5 6 7 8 9
Customer satisfaction scores
Enterprise cost savings
Frequency standard deviation/Frequency Control/
Frequency control index
Other

Page 32
Question 12: How often is there analysis of the corporate KPIs?
The results of the questionnaire demonstrate that most organisations review/analyse their KPIs monthly (Figure
4.16) and all organisations set these KPIs on an annual basis.
Figure 4.16 Periodicity of corporate KPI analyses
Question 13: What operational KPI is linked to corporate KPI Incident Frequency Rate/lost time accidents/lost time
injury rate?
Certain Corporate KPIs are mapped directly to the operational KPI whilst others are more ambiguous and are
therefore interpreted differently by each TSO’s. For example the corporate KPI representing incident Frequency
rate/lost time accidents/Lost time injury Rate is mapped mostly to the same operational KPI as well as Reliability
and Energy Not Supplied, Figure 4.17 which could be argued are all similar KPIs based on the same principles.
Figure 4.17 Operational KPI related with Corporate KPI: Incident Frequency Rate/Lost Time accidents/Lost
time injury rate
Question 13: What operational KPI is mapped to corporate KPI Performance Monitoring?

Page 33
Figure 4.18 Operational KPI related with Corporate KPI: Performance Monitoring
Figure 4.18 demonstrates the operational KPIs mapped from the Corporate KPI Performance monitoring. Most of
TSOs/DSOs/RTUs depend on Demand Forecast and Project Delivery (which could be linked to asset growth) as an
indication of Performance Monitoring.
Question 13: What operational KPI is mapped to corporate KPI Transmission Capability?
Figure 4.19 links Outage Realisation Costs, Constraint Cost and Curtailment of wind generation as the major
Operational KPIs mapped from corporate goal: Transmission Capability.
Question 13: Corporate KPI’s and Operational KPI’s
Figure 4.19 Operational KPI related with Corporate KPI: Transmission Capability
Question 13: What operational KPI is mapped to the corporate KPI Customer Satisfaction Scores
Some TSOs/DSOs/ RTUs have similar Operational KPIs but these are mapped from different Corporate KPIs. An
example (on Fig.4.20) of this is robustness and reliability rates which are mapped from both Corporate KPIs
Incident Frequency Rate/Lost Time Accidents/Lost Time injury rate and Customer Satisfaction Scores. Quality of

Page 34
supply exceedances and customer queries are the primary KPIs linked to Customer Satisfaction Score. It would be
interesting to note how the quality of supply exceedances is measured.
Figure 4.20 Operations KPIs related with Corporate KPI: Customer Satisfaction Scores
Question 13: What operational KPI is mapped to the corporate KPI effectiveness Core Team?
Figure 4.21 displays the operations KPIs associated with the Corporate KPI on the Effectiveness of the Core Team.
About 35% of TSOs/DSOs/RTUs do not map operational KPIs to this corporate KPI.
Figure 4.21 Operations KPIs related to Corporate KPI: Effectiveness Core Team
Question 13: What other operational KPIs are linked to corporate KPIs?
There are a number of other Operational KPIs which are mapped to Corporate KPIs which were not specified.
These operational KPIs are listed in Table 4.2

Page 35
Table 4.2 Operational KPIs associated with Corporate KPIs
Line faults/100km
Telecommunications availability
Unplanned circuit unavailability
Frequency standard deviation/Frequency Control/ Frequency control index
N-1 violation
Plant Performance Index (PPI)
Reportable number of voltage excursions/ Voltage Quality Index/Voltage Control
(DTRP)
Energy Not Supplied per year/ System Minutes Lost < 1/ System Minutes Lost >1
IT/OT availability
Number of safety / switching incidents/ Operations Performance
Question 13 : What IT support monitors performance at Corporate level ?
The IT systems that monitor Corporate KPIs are mainly either Business Intelligence/Data Analysis or a Traffic Light
System. Figure 4.23 demonstrates this.
Question 14: IT Support
Figure 4.23 IT systems monitoring Corporate KPIs
4.4 Operational Goals and KPIs
The Operational goals and KPI’s were divided into six categories (goals) as listed below with specific questions on
which goal and KPIs are mapped by the System Operator to specific categories/goals. The operational categories
(goals) were:
 Grid Security and Reliability
 (Good) Operational Practices
 Operational Safety and Environment
 Economical Operations
 Customer Focus
 Employee Satisfaction and Development

Page 36
Scope of the Questions 15-20
The scope of the Questions 15 to 20 was to provide an overview about the “measurability” of the Companies’
performance in specific activities that attain the TSO/ISO mission. For example, the security of supply and system
reliability are unanimously considered as crucial issues, all the System Operators agree that they have to be
guaranteed in their respective control areas, but it’s not always easy to identify a clear indicator that shows if the
performances are acceptable or not.
In some cases these performances are measurable as, for instance, the Energy Not Supplied (ENS); in other
cases the identification of a synthetic numeric indicator that expresses the power system situation in specific
operational conditions is much more complicated. Some indicators can measure specific issues, but the “global
indicator” that represents the system condition as a whole is maybe impossible, or at least extremely hard, to
determine.
On the other hand, in a deregulated framework as it is the case today, the System Operators must act in the most
transparent way in front of external stakeholders, with particular regard to the National Regulators. Moreover,
without an objective measurement of specific activities and a KPI to be fulfilled, it is impossible to measure the
performance in terms of distance from the desired target and to identify a way for its improvement.
In this perspective, the responses in this subject provide an overview of the current practises in different Power
Systems. The synthesis in the following section is intended as a comparison of the best practices adopted, bearing
in mind that the differences that arise may depend on the different National or International regulatory framework in
which the System Operators act.
Statistical Data and Analysis
In the following paragraph, is the high-level analysis of the responses received for each question according to the
operational targets and KPI’s section.
Question 15: Are there operational performance goals defined in your organization?
Figure 4.24 Defined Operational Performance Goals in Companies
As expected most utilities will have goals that are defined and monitored through KPIs. It is interesting to note that
almost a third of the respondents indicate that their operational goals are not defined through KPIs. This may need
to be investigated in more detail in the future.
Although almost a third of the respondents indicate that their operational goals are not defined through KPIs they
(4) didn’t provide the information that was used instead. Just one of them indicated that other (non-operational)
departments, whose representatives were not involved in the Questionnaire due to time limitations, define some of
the stated goals/KPIs.

Page 37
Question 16.1: Grid Security and Reliability - What operational goals and KPI are explicitly used?
Figure 4.25 Grid Security, Reliability Goals and related operational KPIs
Under the ‘Grid Security and Reliability goal the KPIs used by most are the “Frequency standard
deviation/Frequency Control/Frequency control index”, with “Energy Not Supplied per year/ System Minutes Lost”.
The next KPIs that is used by approximately half of the respondents are “N-1 violations” and then “Incident
Frequency Rate”. The remaining KPIs under the Grid Security goal are used by less than 5 out of the 16
respondents. It could be assumed that if a Grid Operator closely managed the first four KPIs mentioned then the
remaining KPIs would be effectively covered. Yet, taking into consideration the large system incidents in the last
decade, where voltage instability and unavailability of EMS systems were the main contributors to these incidents,
it is interesting to note that voltage stability and IT/OT availability are not a higher priority for many Grid Operators.
Question 16.2: Operational Practices - What operational goals and KPI are explicitly used?
Figure 4.26 Operational Practices Goals and KPIs
The two KPIs with the largest number of responses show that investment in operating planning capacity
anticipating the difficulties faced by operating teams, improve the predictability of events and behaviour of the
system. At the next level KPIs for operation performance and quality of measurement, are important points of
observation. From the identified KPIs it can be seen that many utilities are in the process or have already
implemented some of the ISO standards.

Page 38
Question 16.3: Operational Safety and Environment - What operational goals and KPI are explicitly used?
Figure 4.27 Operational Safety and Environment Goals and KPIs
The security of the power system and concern for the environment remains present and are explicitly measured by
indicators. Of the identified KPIs, the “Number of Safety/ switching incidents/ Operational Performance” can be
seen as the most important measured KPI and is assumed to be directly influenced by the competency level of
operational staff and quality of training received.
Question 16.4: Economical Operations - What operational goals and KPI are explicitly used?
Figure 4.28 Economical Operations Goals and KPIs
The spread of responses indicate that economics remain an important consideration but are addressed in very
different ways, probably adherent to the realities of each surveyed Operator. It is likely that a significant number of
operators surveyed still keep the philosophy of honour reliability in operation despite its cost, in real-time decisions.
Question 16.5: Customer Focus - What operational goals and KPI are explicitly used?
Figure 4.29 Customer Focus Goals and KPIs

Page 39
There is a balance in the responses to customer satisfaction and a similar approach is adopted among all those
surveyed. It remains ideal that results are linked to the end customer’s perception of them.
Question 16.6: Employee Satisfaction and Development - What operational goals and KPI are explicitly used?
Figure 4.30 Employee Satisfaction and Development Goals and KPIs
From responses received it can be seen that there is still a weak link between measuring the results of the
operation and employee satisfaction.
For Question 16.7 a list of other operational goals and KPIs as given by respondents.
The answers received are given in the table 4.3 in the sequel.
Table 4.3 Other KPIs used.
Average System Availability 1
Budget 1
Demand forecast errors/ Demand Forecast Assessment 1
Hours of training 1
IT/OT availability 1
N-1 violation 1
Quality of Supply Exceedances 1
Unplanned circuit Unavailability 1
Other 20
So some ‘’other’’ answers were identified, but just a few provided information about what “other” measurements are
used. We obtained little information with generic answer "other".
In these comprehensive answers there appears some indicators listed above, but they are not linked to the KPIs of
the previous section. It shows the different approaches arising from indicators.
Question 17: How often are the Goals/KPI’s adapted?
Figure 4.31 Frequency of Goal/KPI adaption

Page 40
The high concentration showing annual adaptation of indicators shows a typical concept for KPIs setting with
annual assessment and analysis, covering the whole set of seasonal aspects of human behaviour and, not
disconnected, the load behaviour and system operation.
Question 18: How often are operational level KPIs reported on?
Figure 4.32 Frequency of reporting on Goal/KPI
In addition to indicators adaptation have already shown, the frequency of reporting of indicators shows little
concern with their volatility. The need for analysis of a larger sample predominates, for the results to be considered
as a general conclusion.
Question 19: Is there a public version of the operation level KPI and how often is it updated?
Figure 4.33 Public version of the operation level KPI and frequency of updating
The same observations as submitted to the previous question are valid.
Question 20: Is there a suitable IT support to monitor performance at the operation level?
Figure 4.34 Suitable IT support exist to monitor performance
The search for IT support to facilitate KPIs is a general trend with the high speed of the evolution of computing
resources.

Page 41
4.5 Training goals and KPI’s
SCOPE OF THE QUESTIONS 21-33
The scope of the Questions 21 to 33 (see Annex 1) are to valuate if there is a close relationship between business
performance (both corporate and operational) and those related to the training of operators.
It is interesting to find if the goals and KPIs that the company makes in Corporate level, and on the electrical
system operational level, are reflected in the methodologies and training practices and whether there is a genuine
link between general indicators and operator training practices.
Some other interesting evaluations could be done taking into account the different regulatory systems which the
Companies are subject to.
STATISTICAL DATA AND ANALYSIS
Following the high-level analysis of the responses received for each question in the questionnaire under the
Training level goals and KPI’s.
Question 21: Select training goals you use
The scope of this question is to identify the characteristics of the training goals usually performed. To compare the
answers easily a multiple choice of predefined answers were possible but also ‘Others’ to describe answers that
were not predefined.
The possible (multiple) choices regarding training goals were:
 System Operation Management
 Handling of planned and unplanned outages
 Minimize number of switching mistakes
 Emergency plans
 Black start and restoration
 Coordination and communication with neighbouring network partners (TSO, DSO, PP, ...)
 New functions (e.g. new control centre tools, new market concepts, ...)
 Other
Responses received
Figure 4.35 Training Goals used

Page 42
Some companies have indicated some notes (when chosen other) that are detailed below:
 Theoretical Training: such as academic info sessions, components and equipment, and orientation,
followed by simulation sessions.
 'Continuous learning' subjects, basics in electrical engineering high voltage theory. All topics are related to
practical issues.
 Common training with the neighbouring TSOs.
Training Goals are included as individual’s goals for the year. If selected for advancement a training plan would be
developed specifically for the single operator.
The training of the operators used in companies usually covers the normal and the abnormal situation, inclusive of
the black start and restoration procedures.
The aspects that are not widely covered are the minimization of the switching mistakes and handling of planned
and unplanned outages. Answers regarding communication and coordination with neighbouring network partners
are, of course dependent on the grid structure; those who have more partners on the TSO level answered
positively.
One company defined all goals as individual’s goals for the single operator for the year (choosing no training
goals). Another company chose only one goal: 'System operation management'. The remaining companies
answered more or less all predefined answers with yes.
Generally, most of the above goals are followed by companies, but there are not always defined goals.
Additional training goals were asked in Question 22 (see Annex 1).
Question 22: Specific training goals used? (If not mentioned in Question 21)
Question 22 was without selection of answers. Six companies answered as followed others haven't describe
specific training processes:
 New Operators will have an individual training plan established for initial training. Any subsequent
advanced training will also include an individual training plan. A list of tasks required for each operator
position has been developed. New Operators are provided with a Training Map/Plan that provides a
sequence for the specified classroom training, OJT, assessments and finally qualification for each
particular position. The completion of each step in the qualification process is documented and progress is
monitored by the training group. The operators are given a pre-assessment exam when they are first
assigned to operations to help determine their base knowledge and the qualification timeline is adjusted
accordingly.
 A training program is a planned, organized sequence of activities designed to prepare individuals and
teams to perform their jobs and to maintain or improve their performance on the job. It describes the
training as consisting of an initial program and a continuous program of training goals. The training has to
be permanently adapted to the operational evolutions. Dispatchers in contact with neighbouring control
areas shall have sufficient knowledge of English and operational terms to carry out their tasks.
 There are no separate soft skills trainings but knowledge and soft skills are combined in simulation
training. So most of the trainings have also training goals related to soft skills.
 Defined knowledge level about a power system, English, personal development and team building
 Additional training given to improve the understanding of operational tasks for field personnel
 Internal OTS
Interestingly, four of the six answers mentioned communication with others (for example common language
English) and also understanding of partners, for example the understanding of operational tasks with field
personnel. One company will start up soft skill training in the future and combine with simulation training. In future
soft skill training has to be pointed out in more detail.
Question 23: Select training KPI's you use
This question should focus which are the main targets defined for training

Page 43
The possible (multiple) choices were:
 Percentage or number of successful certification
 Percentage of trainees meeting planned training schedule
 Percentage of passing certification for the first time
 Average training time to meet certification
 Average training days per operator per year
 Measure of company-external trainings
 Qualitative rating of all training activities, based on evaluations
 Operator performance over time (additional input about quality of training and needed further training)
Responses received
Figure 4.36 Training KPI’s used
The results show that there is wide variation in KPIs used to measure training performance. The KPI based on a
qualitative rating is used by most companies to define training success.
Most of the companies (except one) use KPI's for training. But the combination of used KPI is different and can be
split in two main categories. One combination is the 'percentage of trainees meeting planned training schedule' and
the 'Operator performance over time', the second combination is vice versa in that all KPI's are used except the two
of the previous combination. Answers of combination two covers the answers of combination one therefore it is not
necessary to have extra KPI's.
Inter-TSOs trainings with neighbours were defined as one of the following three:
 Cross visits between neighbouring TSOs dispatchers,
 Common training workshops
 Common simulation sessions
One company answered as followed describing specific training processes:
 All qualified operators will attend Continuing Training. New Operators will have an individual training plan
established for initial training and any subsequent advanced training. The company employs a systematic
approach to training utilizing the ADDIE process. A list of tasks required for each operator position has
been developed. New Operators are provided with a Training Map/Plan that provides a sequence for the
specified classroom training, OJT, assessments and finally qualification for each particular position. The
completion of each step in the qualification process is documented and progress is monitored by the

POWER SYSTEM OPERATOR PERFORMANCE: CORPORATE, OPERATIONS AND TRAINING GOALS AND KPI’S USED

POWER SYSTEM OPERATOR PERFORMANCE: CORPORATE, OPERATIONS AND TRAINING GOALS AND KPI’S USED

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to POWER SYSTEM OPERATOR PERFORMANCE: CORPORATE, OPERATIONS AND TRAINING GOALS AND KPI’S USED

Similar to POWER SYSTEM OPERATOR PERFORMANCE: CORPORATE, OPERATIONS AND TRAINING GOALS AND KPI’S USED (20)

More from Power System Operation

More from Power System Operation (20)

Recently uploaded

Recently uploaded (20)

POWER SYSTEM OPERATOR PERFORMANCE: CORPORATE, OPERATIONS AND TRAINING GOALS AND KPI’S USED