This document discusses meaningful security metrics for various stakeholders. It recommends metrics that measure policy compliance, control maturity, and value at risk for CIOs. For operations managers, it suggests metrics that track systems outside of SLAs and security incidents breaching SLAs. For CISOs, suggested metrics include value at risk, compliance, and annual risk reduction compared to spending. For CEOs and boards, total exposure and unmanaged risk are recommended metrics. It also provides characteristics for effective security metrics and metrics for evaluating the metrics themselves.

1. Meaningful and useful Security metrics Vladimir Jirasek About.me/jirasek 5st Oct 2011

2. About me Security professional (11 years), current work at Nokia as Enterprise Security architect Founding member and steering group member of (Common Assurance Maturity Model) CAMM (common-assurance.com) Director of Research, CSA UK & Ireland I love reading books: thrillers (Clive Cusler) and business management (Jo Owen)

3. I will cover three topics today Information Security Model Metrics for CIO Metrics for Operations manager Metrics for CISO Metrics for CEO and the Board

4. Security model – business drives security Feedback: update business requirements International security standards Governance Input Line Management Security management Correction of security processes Laws & Regulations Product Management Process framework Policy framework Metrics framework Define Information Security Metrics objectives Information Security Processes Information Security policies Program Management Compliancerequirements Measured by Mandate Input Inform Information Security standards Risk & Compliance IT GRC Business objectives Technology People Services Assurance Information Security guidelines Define External security metrics Business impact Auditors Measure security maturity Execute security controls Define security controls Define Security management Business & information risks Security intelligence Security Services Security Professionals Input Security threats

5. Security metrics characteristics Measurable Objective Quantitative (ideally) Meaningful With KPIs attached – know what is good and bad Linked to business objectives – money speaks

6. Metrics for CIO – (1) Policy compliance and control maturity

7. Metrics for CIO – (2) Value at risk* Input Asset values Maturity of controls System weaknesses Threat information Output – most likely (probability distribution) £ value of total exposure that IT organisation is exposed to Inspiration in BASEL II Work in progress * Eq most likely Total Exposure

8. Metrics for Ops manager The morning dilema: “Can I have a coffee or is there something urgent to fix?” Suggested metrics: A number/percentage of systems outside SLA for fixing security weaknesses (both patches and configuration errors) – details of highly critical offenders – sorted by value at risk Security incidents that resulted in breached SLA (SLA is both time and £ value) And of course: Value at Risk Quiz: Is “A number of critical vulnerabilities good metric?” Answer: Not on its own!

9. Metrics for CISO Gartner: by 2014 IT GRC and eGRC will merge in 70% of organisations. Likely head: CISO Relevant metrics: Value at Risk – includes IT and other departments Compliance matrix ( same as for CIO) Annual risk reduction - Difference between VaR now and last year compared to money spent

10. Showing value for money End year review: We have spent more than the risk reduction but there were no incidents! VaR can also increase with new business processes and changes in regulatory and threat landscape.

11. Metrics for CEO and board Total exposure (£) = Value at Risk indicator Unmanaged risk = likelihood there are risks that we do not know about = inverse of eGRC maturity

12. How do I know I have good metrics – metrics of metrics Decision effectiveness approach % of important management decisions that can be or have been influenced by double learning (i.e. revision and refinement of targets, measures, criteria, etc.) Investment approach % of security metrics costs for “exploratory/testing” vs. total metrics cost Speed Cycle time from “Sense” to “Respond” for changing security metrics and management procedures. % of metrics that are collected and calculated automatically Cost Cost of changing security metrics and management procedures as % of total security management costs. Error % of security metrics that do not tie to any decisions or decision processes (over-shoot) % of decisions that have inadequate metrics support (under-shoot) % of metrics which have significant number of false signals

13. Summary Metrics need to include monetary value otherwise the business leaders will not understand why the metrics are collected and presented Security (and GRC in general) are here to keep the company risk at acceptable level – that needs to be measured Link security metrics to policy which is linked to business objectives Boards do not like “un-managed risk” Measure the metrics