Measuring Success - Security KPIs

Measuring Success Security Metrics and KPIs Meaningful InfoSec Program Measurements Harry Contreras - CISSP, Six Sigma Phoenix, AZ www.company.com

Why measuring your InfoSec Program matters Topical overview of this presentation material • What are metrics and measurements? • Why use process control methodologies to measure security programs? • What does this information tell us? • What are the benefits? • KPI’s and how they are derived • How can Information Security Programs be effectively measured? • Translating business risks to metrics • How effective is your InfoSec program? • Resources – The where and how to get additional information • Summary • Questions and Answers Measurement - Best practice in management and control methodologies Presentation to ISSA – Phoenix, AZ – April, 2009 1

Why measuring with metrics matter Why measuring your InfoSec Program matters Security is a process • So why are we not using process control methodologies to measure and advance our security programs and initiatives? As security practitioners we should be measuring the value of information security programs and demonstrating the continuing maturity of the organization. Why we measure • Integral to process controls and governance programs • Integral to measuring deficit areas and where to focus for improvements • Integral to overall information security program success Someone once said… You don’t know what you don’t know… Presentation to ISSA – Phoenix, AZ – April, 2009 2

Why measuring with metrics matter Security Metrics as Established Industry Best Practice One example from industry best practice for security controls ISO17799/2005 4. Risk assessment and treatment 4.2 Treating security risks d) cost of implementation and operation in relation to the risks being reduced, and the remaining proportional to the organization’s requirements and constraints e) the need to balance the investment in implementation and operation of controls against the harm likely to result from security failures In the information security practice, the resulting actions of our control processes is to continuously improve the Company’s business risk profile. Presentation to ISSA – Phoenix, AZ – April, 2009 3

Why measuring with metrics matter Why measuring your InfoSec Program matters Measure to Manage • Converting reporting inputs into meaningful outputs • Compile data and develop representations of the information Metrics must be meaningful to the company • Every company is different • There is no one metric to fit all Develop a Security Scorecard • Regular and consistent measurements from baseline numbers • Require metrics and regular reporting from security service providers • Performance assessment to service level objectives & agreements Someone once said… You don’t know what you don’t know… Presentation to ISSA – Phoenix, AZ – April, 2009 4

Assessing meaningful metrics to report Strategic Metrics for Information Security - P. Lindstrom, Sept, 2008 Presentation to ISSA – Phoenix, AZ – April, 2009 5

Why measuring with metrics matter Why measuring your InfoSec Program matters Measure to Manage • Metrics are measures used to indicate progress or achievement • Measurements are a quantitative assessment of a circumstance Metrics can be improved • Measurements do not need to be as they can be observations of a state • Primarily inputs and triggering events from an external condition E.g. number of external scans against the company firewall, External events that are reported by not influence by internal actions Presentation to ISSA – Phoenix, AZ – April, 2009 6

Measurements and Metrics – Good and Bad Controlled and Uncontrolled Events Strategic Metrics for Information Security - P. Lindstrom, Sept, 2008 Presentation to ISSA – Phoenix, AZ – April, 2009 7

Why measuring with metrics matter Why measuring your InfoSec Program matters What metrics and measurements provide and how not to use them • Provide basis for continual improvement • Provide strategic intelligence for management • Provide fiscal overview for aligning spend to company priorities and business goals Don’t measure everything • Metric or Measurements – Holistic vs. minutiae • Not for “reward and punishment” - Participants will learn how to “game” the system What does leadership want to know? • What is our level of risk? • How strong is our security program? • Are we maintaining appropriate cost control? Presentation to ISSA – Phoenix, AZ – April, 2009 8

Why measuring with metrics matter Why measuring your InfoSec Program matters Measure to Manage • Converting reporting inputs into meaningful outputs • Compile data and develop visual representations of the information Reporting Actions • Regular and consistent measurements from baseline numbers • Require metrics and regular reporting from security service providers • Performance assessment to service level objectives & agreements Rationalization of metrics • Align with industry recognized statistics to gauge your business risk profile • E.g., CSI Annual Computer Crime and Security Report Prioritization for Actions • Budgeting for Capex and Opex • Present factual representation of security state with measurements Reassessment Actions • Measurable difference in business security state can be identified Presentation to ISSA – Phoenix, AZ – April, 2009 9

Security Program Information–Decision–Action Cycle AKA, Plan, Do, Check and Act (PDCA) Actions Data Apply mitigating security Monitor threat horizon, review new controls or changes to technologies, develop services delivery portfolio Effectiveness measurements. Interpretation Assess, R&D, review security metrics, Benchmarking, ROSI analysis and Key Performance Indicators Continuous improvement through repeatable process controls Presentation to ISSA – Phoenix, AZ – April, 2009 10

Key Performance Indicators - KPIs When identifying KPIs to set as measurement targets, select ones that you have control over and can improve. Presentation to ISSA – Phoenix, AZ – April, 2009 11

Developing KPIs Key Performance Indicators – 10 Critical Characteristics KPIs reflect strategic value drivers KPIs are defined by “executives” KPIs cascade throughout an organization KPIs are based on corporate standards KPIs are based on valid data KPIs must be easy to comprehend KPIs are always relevant KPIs provide context KPIs empower users KPIs lead to positive action Key Performance Indicators are metrics, but not all metrics are key performance indicators. Presentation to ISSA – Phoenix, AZ – April, 2009 12

Thought Process Map Thought Process Map for Security Metric Development Process Identify possible Related to Voice of Customer Is it Is it Start metrics targets What measure? Input quantifiable? repeatable? Yes Are metric Data extrapolation Key Performance Related to Data sources Targets Processes Indicators Cost dollars? identified? Viable? developed Derived No Measures KPIs added to Interpret Quantifiable Voice of Customer Based Executive Metric Change End Input Actions & Dashboard results Results Decisions Continuous improvement through repeatable process controls Presentation to ISSA – Phoenix, AZ – April, 2009 13

Why measuring with metrics matter Measurement and Analysis - Examples Assess the viability of your target measurements with meaningful criteria Analysis Target What does Customer Source Quantifiable Repeatable Derived it Measure Measures Cost Mean time to patch Exposure On time Patching Yes Yes $$ application window (SLO) System Content filtering event Effectiveness Cost SOC Yes $ counts Percent of un-patched Risk index Patching Yes Yes systems to asset inventory System AV events detected and Effectiveness Reliability AV service Yes Yes $$ cleaned Mean time to AV control file Exposure On time AV Service Yes Yes $ update window (SLO) Average historical spend per Historical Yes No $$$$ InfoSec Incident records IDS incident reporting rate IDS Yes $ system SPAM messages Effectiveness Customer Service $$ suppressed Sat Records Presentation to ISSA – Phoenix, AZ – April, 2009 14

Methods to derive Security Program Metrics Strategic Metrics for Information Security - P. Lindstrom, Sept, 2008 Presentation to ISSA – Phoenix, AZ – April, 2009 15

What actions do you take with your metrics? Why measuring your InfoSec Program matters There are costs associated with controls that extend beyond the implementation of those controls. • How long will the control be effective? • Is the cost of the control reasonable, relative to the value of the asset? How can these numbers relate? • Align with any in-Company compliance programs • Align with other recognized industry statistics • Annual industry published reports • E.g. CSI’s Annual Computer Crime and Security Report The company internal valuation process, each company’s approach is different. • Process through Business Governance path • Internal business financial valuation processes are different • Your mileage will vary Presentation to ISSA – Phoenix, AZ – April, 2009 19

Why measuring your InfoSec Program matters Develop a Security Program Scorecard • Company risk index (one of may options) • IT Security metrics and KPIs • Measure InfoSec program effectiveness • Regular and consistent measurements from baseline numbers • Require metrics and regular reporting from security service providers • Performance assessment to service level objectives & agreements • Represent this information in a visual form, perhaps an information security dashboard for leadership to monitor In today’s information security practice, consider the aspects of combining reporting information in a “converged” security program for your company. Presentation to ISSA – Phoenix, AZ – April, 2009 20

Why measuring your InfoSec Program matters Identify – The Reality vs. The Perceived business state Business risk profile development for measuring and reporting • Converting reporting inputs into meaningful outputs • Compile data and develop representations of the information Derive an “overall” company risk index or set of indicators • Conducting compliance measurements both internally and externally • Deriving decision support and governance controls • Performance assessment to service level objectives & agreements In the information security practice, the resulting actions of our control processes is to continuously improve the Company’s business risk profile. Presentation to ISSA – Phoenix, AZ – April, 2009 21

Metric and Measurements vs. Business Value What tips the scale in the assessment of business value? Pros Cons • Costs associated with metrics • Provides business baseline • Ongoing activity • Aligns actions with results • Staff overhead • Insight for Governance decisions • Many variables • Visual indicators for: • Information compilation • Effectiveness measures • Disparate recording instances • Risk profile analysis • Multiple inputs • Cost analysis • External influences • Compliance profile • Analysis paralysis • Reality vs. perceived is revealed • Visibility of poor performance The overall importance of IT Metrics are the value to the business in representing the state change associated with the measured activities (Good & Bad results) Presentation to ISSA – Phoenix, AZ – April, 2009 22

What actions do you take with your metrics? Why measuring your InfoSec Program matters A repeatable process with consistent results Analysis Actions • Converting inputs into meaningful outputs • Compile data and develop representations of the information Reporting Actions • Reports, reports reports… • Feed into management dashboards • Presentation to leadership Rationalization Actions • Risk impact assessment • Process through Business Governance path Prioritization Actions • Budgeting for Capex and Opex • Allocation of time and personnel for changes Reassessment Actions • The cycle of continuous improvement Presentation to ISSA – Phoenix, AZ – April, 2009 23

Why measuring with metrics matter Why measuring your InfoSec program matters Who is watching…? External Observers IT Audit practices Compliance assessment organizations *Standard & Poor’s (S&P) Enterprise Risk Management (ERM) Analysis for Credit Ratings of Non-Financial Companies * Request for Comment (November, 2007) S&P has proposed a rating criteria for an Enterprise Risk Management assessment approach. - How well, or even if companies are proactively and effectively managing their business risks. Assessment of a Company’s approach and maturity in this critical business area. Presentation to ISSA – Phoenix, AZ – April, 2009 24

If only it was this easy… Visualize your information security dashboard here… Presentation to ISSA – Phoenix, AZ – April, 2009 25

Resources – Helpful slides (One of Two) These important references will aid in developing a security metrics program Information Week Analytics – Governance Vs. Success: Models and Metrics December, 2008 http://informationweekanalytics.com/ Available to companies via the publication’s online hosting of this content. Microsoft – Security Risk Management Guide v1.2 March 15, 2006 Microsoft Corporation. All rights reserved. Download and On-line Locations for the Security Risk Management Guide Specifically sections: Measuring Program Effectiveness, Conducting Decision Support - Download Center: http://go.microsoft.com/fwlink/?linkid=32050 - TechNet online: http://go.microsoft.com/fwlink/?linkid=30794 ISO/IEC17799/2005 - Information Security Standard - ISO/IEC 13335-3 Guidelines for the Management of IT Security http://www.iso.org/iso/home.htm Information Systems Security Association - (ISSA) • The Use of ROI in Information Security – by Luther Martin (See Resources – ISSA Journal, Nov 2008) • Security Metrics – Hype, reality and value demonstration – by Aurobindo Sundaram (ISSA Journal, May 2008) • Ways to Determine or Prioritize Security Initiatives – by Matt Ege (ISSA Journal, Jan 2009) • http://www.issa.org/ These are just a few of many additional resources to search in this information repository. CSO Online – The Security Metrics Collection, October 27, 2008 Refer to the Security Leadership section for Metrics and Budget http://www.csoonline.com/ Presentation to ISSA – Phoenix, AZ – April, 2009 26

Resources – Helpful slides (Two of Two) These important references will aid in developing a security metrics program SearchSecurity.com A TechTarget online publication Refer to the Topics section for Information Security Management http://www.searchsecurity.com/ SearchFinancialSecurity.com – A TechTarget online publication • Strategic Metrics for Information Security at Financial Services Firms – P. Lindstrom, Sept, 2008 Refer to the Management Strategies section for additional information http://searchfinancialsecurity.techtarget.com/ International Information Security Systems Certification Consortium - (ISC2) • Why Security Metrics Must Replace Traditional Risk Analysis Methodologies – by Robert Hudock, Mar, 2008 Available to ISC2 registered members via the organization’s online hosting of this content. www.ISC2.org Locate in the ISC2 Journal Archives Security Metrics: Replacing Fear, Uncertainty and Doubt Author, Andrew Jaquith – 336 Pages © 2007, Addison-Wesley Professional Publications. Metrics Management Toolkit - Implementing Metrics Management Guide, Metrics spreadsheet, Project WBS, 125+ predefined templates © 2008, Unified Compliance Framework Inc. http://www.unifiedcompliance.com/ Located in the IT Impact Zones / UCF Toolkits offerings section. Presentation to ISSA – Phoenix, AZ – April, 2009 27

On a final note… FYI - For Information Security Professionals At this year’s RSA Conference 2009 in San Francisco, CA There will be six (6) separate presentations covering security metrics, measuring security effectiveness and data driven C-Level decision making approaches featured this year. Presentation to ISSA – Phoenix, AZ – April, 2009 28

Summary “There are risks and costs to a program of action. But they are far less than the long-range risks and costs of comfortable inaction.” John F. Kennedy With effective security measurements, risk identification, assessments and mitigation approaches businesses can benefit with the following results. • Competitive advantage • Security • Efficiency • Resilience • Confidence Presentation to ISSA – Phoenix, AZ – April, 2009 29

Measuring Success Security Metrics and KPIs Meaningful InfoSec Program Measurements Harry Contreras - CISSP, Six Sigma Phoenix, AZ www.company.com

Measuring Success - Security KPIs

by H Contrex on Apr 02, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 13

Statistics

Measuring Success - Security KPIs — Presentation Transcript