First let’s agree to the definition and rationale for Metrics and their true intent.
Asking some basic questions will help determine the direction of the program.
Metrics can be used in a variety of ways – it is important to understand, evaluate and decide how the metrics need to be used before launching a metrics program.
From a management (C-level, VP and AVP) level, why do it if we can’t prove its value or the expenditure?
Let’s look at a few qualities that will make our data useable and will provide the answer to cost and value.
These are some of the elements that most Information Security teams use as a basis for metrics reports – a lot of teams report on all of these and others as well. But measuring these may not provide the value that management wants or needs. Some of the data is back-end data that will only be meaningful to the InfoSec team – it’s important to begin the process of separation, evaluation, presentation and relevance to the intended audience(s).
We just looked at a list of data points that can be measured and tracked – for each one that is selected, we should be able to know the answers for the What, Who and Why and then set them by priority.
*While 93.4% looks good on the surface and the trend seems upward overall, Without comparison points, those numbers don’t provide effectiveness over time, how many virus’ were known, how many were unknown/new, how quickly they were caught, where they originated, etc.
Once we determine which data points are relevant and have added at least 2 data points, we then have to look at how we present the reports – visually pleasing – and apply interpretation to those metrics. It’s important to present information in such a way that the audience does not have to try to figure out on their own what the significance of the result shows – and the benefit of this is that now we can have actionable results.Example for hiring: We only have a 50% improvement from last year at this time and that means that we are behind schedule of wanting to be 100% complete by years’ end. What we need is 2 more headcount based on requirements for implementation as agreed in the IRMAC meeting and supported by senior management. Going forward we should consider hiring additional staff before setting a tight schedule.Example for positive reinforcement: We have a 50% improvement from last year at this time and that means that we are ahead of schedule of wanting to be 100% complete by years’ end. What we need is to continue this program based on requirements for implementation as agreed in the IRMAC meeting and supported by senior management. Going forward we should consider doing things in this same manner since it is so effective.
Security Metrics Program
SECURITY METRICS A presentation developed by Cydney Davis, Senior Technical Write
What are Metrics?2 A method which facilitates decision-making and improved performance and accountability through collection, analysis and reporting of performance- related data. Information Security metrics must be: • based on Information Security performance goals and objectives • useful for reduction and management of risks • readily obtainable and replicable • useful for tracking performance and directing resources • able to yield quantifiable information
What is our Mission/Goal?3 It is critical that we use metrics that are relevant to our organization and to the mission we are measuring. But first, we have to determine: • Where we are (Baseline) • Where we are going (End Goals) • Who/what relies on us? (Users/Management) • What do they need/expect? (Reports/Assurance) • What are we trying to prove? • What are we trying to solve? • What are we trying to improve?
How can we use Metrics?4 Communicate Performance Drive Performance Improvement Measure Effectiveness of Security Controls Help Diagnose Problems Provide Effective Decision-making Support Increase Accountability Guide Resource Allocation Demonstrate the state of compliance Facilitate Benchmark Comparisons
Metrics can help determine:5 • the number of resources it takes to accomplish security goals • justifiability for financing new security measures • If the company is getting its money’s worth • If the company is managing risk appropriately* • what Information Security needs to do to improve Security ˉ administration/processes/procedures/policies/person nel/enhancements/technology/etc.) • where we are with comparisons to peers regarding to standards, best practices, execution and results of security measures *The residual risk that a company is willing to take based on; business needs, budget limits, industry regulations/requirements and other criteria.
Executive Focus7 “The heart of it is that if a business process cannot be measured in one way or another, we likely ought to cast it off as wasted effort.” Comment from a CEO to an anonymous Information Security Profession Translation: Why do it if we can’t prove/justify its value? (time, money, effort, results and actions)
Good Metrics Guidelines8 • Consistently Measured • apples to apples/same time same place • Cheap to Produce (Time-wise) • Yield Quantifiable Information • Contextually Specific – who • Expressed using at least 2 units of measure or data po
Metrics Program Success9 Criteria Identify incident trends important to key senior managers, stakeholders and to the InfoSec Mission from a management perspective.* Provide consistent information that adds value and is actionable by: • Tracking changes on a consistent basis. • Focusing on whats important in our business • Developing a few value indicators that we can track with a high degree of reliability • Doing some service benchmarking with our peers. *This is the first and most important decision
Basic Information Security10 Measures Anti-malware Firewalls Asset Management Intrusion Anti-SPAM Patch Detection Management and Prevention Vulnerability Unified Threat Application Management Management Security Scanners Databases Website Statistics Network Access Control System Integrity Operating Data Leakage Checking Systems Protection Configuration Secure Web Web Application Hardening Gateways Firewalls Mobile Data Media Sanitation Storage Protection Encryption
Formula for Deriving True11 Meaning WHY we need WHAT we need WHO we are to measure it to measure measuring it for • Financial DATA • C-Level • Governance DATA • Board of Directors • Legal DATA • Marketing Releases • Regulatory DATA • Industry Report • Directive DATA • General Staff Determine how the information will be analyzed, interpreted and used!
12 “Good metrics facilitate discussion, insight and analysis...”
Metrics Program - Components13 Program Component Define the metrics program goal(s) and objectives Decide which metrics to generate Develop strategies for generating the metrics Establish benchmarks and targets Determine how the metrics will be reported Create an action plan and act on it Establish a formal program review/refinement cycle
High Level Process Steps14 Obtain management input, agreement and support for the implementation of a strong metrics program. Review our organization’s mission statements, policies, plans, procedures, goals and objectives, and assess them against legislative and regulatory requirements, as well as against effectiveness goals. Describe how we will achieve company and department goals List milestones, dates and quantifiable objectives against which to map progress. Select appropriate, quantifiable effectiveness metrics to indicate baseline, interim and final success. Gather the metrics. Analyze and present the results to management and key stakeholders. Recommend that management make decisions based on the metrics, and plan the execution of these decisions. * Metrics are often referred to as “decision support.” Evaluate the outcome of decisions against goals. This should be done from a perspective of *The real value of a metrics program
Good metrics are those that are17 SMART; • Specific • Measurable • Attainable • Repeatable, • Time-dependent Truly useful metrics indicate the degree to which security goals are being met – and they drive actions that need to be taken to improve our overall security goals.
Metrics? Or Just Numbers?18 Exhibit A - This set of numbers can give us a sense of the overall health of anti-virus defenses and can show trends over time; but the information is not actionable in any way and will not serve as a meaningful diagnostic tool. SO WHAT??? = False sense of security without more knowledge
Good Metrics = Numbers with Relevance19Exhibit B displays the same measurements as Exhibit A. By drilling down into the data we can begin tounderstand which locations are struggling with this activity. This in turn will help us choose where to focus inorder to improve the performance of our organization. This kind of actionable intelligence is valuable and itcan really drive performance improvement and provide information that is actionable to a productive end. Example Metrics showing RELEVANCE Percentage of computers with current anti-virus definitions City A 99.4 % City B 94.7 % City C 89.8 % 50 % 55 % 60 % 65 % 70 % 75 % 80 % 85 % 90 % 95 % 100 %
Good Metrics = Actionable20 Percentage of computers with current anti-virus definitions CITY A 99.4 % City B 94.7 % City C 89.8 % 50 % 55 % 60 % 65 % 70 % 75 % 80 % 85 % 90 % 95 % 100 %Example Question: Why is one location so much farther behind in implementation?Possible Reasons: Understaffed Limited Bandwidth More staff traveling that previous yearsPossible Actions: Hire additional staff Share resources if the implementation MUST be done by xxx date Set different schedules for each location for future projects
Presenting and Interpreting Data21 Reports Visually Appealing Visually Appealing Interpreted and Actionable _______% improved _______% improved from _______ and that means _________ . What we need is ______ based on requirements for __________ . Going forward we should consider doing ___________ .
22 Measuring for value not numbers Examples to work with Defining, refining and Interpreting data/results for the intended audience
EXAMPLE Metric : Baseline Defenses Coverage (Antivirus, Antispyware, Firewall, etc)23 Measurement of how well we are protecting our enterprise against the most basic information security threats. Just Numbers: ________ % What would an additional relevant value be that we can use to have SMART data? Metrics: ________ % Increase since (prior month/inception/year over year/etc.) Device Type Location Length of time it took to detect
EXAMPLE Metric : Legitimate E-Mail Traffic Analysis24Legitimate e-mail traffic analysis is a family of metrics including incoming and outgoingtraffic volume, incoming and outgoing traffic size, and traffic flow between our company andothers.By monitoring legitimate e-mail flow over time, we can learn where to set alarm points.Numbers:Compare the amount of good and junk e-mail that we are receiving____ percent good____ percent junkWhat would an additional relevant value be that we can use to have SMART data?Metrics ____ percent good ____ percent junk Quarterly/Annually/Since inception/Current Month Since adding the _________ criteria Received from _________ types/organizations Sent During ____________ (AM/PM – Holidays , etc.) Junk Detected Quicker _______ (first time/second time)
Conclusion25 By presenting information in a sufficiently granular way we can inject business relevance into the exhibits. Producing a benchmark is also a powerful approach to performance improvement. Percentage of computers with current anti-virus definitions City A 99.4 % City B 94.7 % City C 89.8 % 50 % 55 % 60 % 65 % 70 % 75 % 80 % 85 % 90 % 95 % 100 % Frequently this level of visibility will spark a competitive fire in those being measured. Professional pride will drive most people to make sure they are found among the high performers on your report.