2. • Source: Security Metrics Guide for
Information Technology Systems, July
2003 http://csrc.nist.gov/publications/
nistpubs/800-55/sp800-55.pdf
3. Components of a Metrics Program
• Strong upper-level management support
• Practical Security policies and procedures
• Quantifiable performance metrics
• Results-oriented metrics analysis
4. Roles
• Head of the Agency
• Chief Information Officer
• Agency IT Security Program Manager
• Program Manager/System Owner
• System Security Officer
5. Purpose of metrics
• Metrics are tools designed to facilitate decision
making and improve performance and
accountability through collection, analysis, and
reporting of relevant performance-related data.The
purpose of measuring performance is to monitor
the status of measured activities and facilitate
improvement in those activities by applying
corrective actions, based on observed
measurements.
6. Characteristics
• IT security metrics must be based on IT security
performance goals and objectives.
• IT security metrics must yield quantifiable
information
• Data required for calculating metrics must be
readily obtainable
• Metrics need to provide relevant performance
trends over time and point to improvement actions
that can be applied to problem areas.
7. Metric Types
Stages of Development (Level of maturity)
Level 1 Level 2 Level 3 Level 4 Level 5
Policy Procedures Procedures P&C P&C
Developed Developed & Controls Tested Integrated
Implemented
Types of Metrics
Goals ObjectivesImplementation Effectiveness Impact
Defined Identified & Efficiency
of security plans
8. Illustration: Level 4 and 5 Metrics
• Computing the percentage of crackable passwords within a
predefined time threshold will validate the effectiveness of
an organization’s password policy by measuring the length
of time required to break policy-compliant passwords.
• The impact metrics would quantify incidents by type (e.g.,
root compromise, password compromise, malicious code,
denial of service) and correlate the incident data to the
percentage of trained users and system administrators to
measure the impact of training on security.
9. Metric Detail Form
• Performance goal
• Performance objective
• Metric
• Purpose
• Implementation evidence
• Frequency
• Formula
• Data Source
• Indicators
10. Performance Goal
• State the desired results of implementing
one or several system security control
objectives/techniques that are measured by
the metric.
11. Performance Objective
State the actions that are required to
accomplish the performance goal. Multiple
performance objectives can correspond to a
single performance goal.
12. Metric
• Define the metric by describing the
quantitative measurement(s) provided by
the metric. Use a numeric statement that
begins with the words “percentage,”
“number,” “frequency,” “average,” or other
similar terms.
13. Purpose
Describe the overall functionality obtained by
collecting the metric. Include whether a metric
will be used for internal performance measurement
or external reporting, what insights are
hoped to be gained from the metric, regulatory or
legal reasons for collecting a specific
metric if such exist, or other similar items.
14. Establishing Performance Targets
• The mechanics of establishing performance targets differ
for implementation metrics and the other three types of
metrics (effectiveness, efficiency, and impact).
• For implementation metrics, targets are set to 100 percent
completion of specific tasks.
• Setting performance targets for efficiency, effectiveness,
and impact metrics is more complex, because these aspects
of security operation do not assume a specific level of
performance.
• Management will need to apply qualitative and subjective
reasoning to determine appropriate levels of security
effectiveness and efficiency and to use these levels as
targets of performance for applicable metrics.
15. Feedback Within Metrics Development Process
• For example, if a security policy defines a specific
password configuration, compliance with this policy could
be determined by measuring the percent of passwords that
are configured according to the policy.
• This measure addresses the level of security control
implementation. It is assumed that configuring all
passwords according to the policy will significantly
reduce, if not eliminate, system compromises through
broken passwords.
• To measure effectiveness of the existing password policy
implementation, the percent of crackable passwords (by
common password-breaking tools) could be identified.
16. Identification and Authentication
• Are users individually authenticated via passwords,
tokens, or other devices?
• Question: Are vendor-supplied passwords replaced
immediately?
• Metric: Percentage of systems without active vendor-
supplied passwords
• Are access controls enforcing segregation of duties?
• Question: Does the system correlate actions to users?
• Metric: Percentage of unique user IDs
17. Logical Access Controls
• Do the logical access controls restrict users to authorized
transactions and functions?
• Question: Is access to security software restricted to
security administrators?
• Metric: Percentage of users with access to security
software that are not security administrators
18. Logical Access Controls (Contd.)
• Are there logical controls over network access?
• Question: Are insecure protocols disabled?
• Metric: Percentage of systems running restricted
protocols
19. Audit Trails
• Is activity involving access to and modification of
sensitive or critical files logged, monitored, and possible
security violations investigated?
• Question: Does the audit trail provide a trace of user
actions?
• Metric: Percentage of systems on which audit trails
provide a trace of user actions
