Metrics: Optimizing Security Operations Performance Dr Anton Chuvakin Chief Security Strategist November 2005

Metrics: Optimizing Security Operations Performance

Dr Anton Chuvakin

Chief Security Strategist

November 2005

WARNING! This is an old presentation; I am publishing it in the hopes that it will be useful for somebody. I do not necessarily share all the views I held back in 2005! And I ramble less now 

This is an old presentation; I am publishing it in the hopes that it will be useful for somebody.

I do not necessarily share all the views I held back in 2005!

And I ramble less now 

Agenda Catalysts for security performance metrics Why security metrics are essential Types of metrics and what to measure Challenges surrounding metrics The metrics lifecycle

Catalysts for security performance metrics

Why security metrics are essential

Types of metrics and what to measure

Challenges surrounding metrics

The metrics lifecycle

Catalysts For Security Performance Metrics Enterprise compliance and governance initiatives Need to manage security as a strategic business process Pressure to demonstrate the efficiency of security technology and related efforts

Enterprise compliance and governance initiatives

Need to manage security as a strategic business process

Pressure to demonstrate the efficiency of security technology and related efforts

Managing Information Security Strategically Compliance has shifted mindset from devices to process Focus on continuous optimization Conduct security assessment Define policies, process, metrics, and baselines Collect data; measure results against baselines over time Identify process and policy weaknesses Refine policy, process, and metrics

Compliance has shifted mindset from devices to process

Focus on continuous optimization

Conduct security assessment

Define policies, process, metrics, and baselines

Collect data; measure results against baselines over time

Identify process and policy weaknesses

Refine policy, process, and metrics

Creating a Culture of Measurement: Recommendations Define security policies and relevant metrics via security assessment Capture all relevant security operational data Generate reports off the data to measure success and identify performance gaps Integrate security performance management data with enterprise compliance & governance initiatives

Define security policies and relevant metrics via security assessment

Capture all relevant security operational data

Generate reports off the data to measure success and identify performance gaps

Integrate security performance management data with enterprise compliance & governance initiatives

Why Security Metrics? You need to know if… your security strategies are successful you require more resources there are in-efficiencies you are compliant “ You can’t manage what you can’t measure”

You need to know if…

your security strategies are successful

you require more resources

there are in-efficiencies

you are compliant

Types of Metrics Technical eg: Percentage of systems with virus protection Process eg: Average virus incident resolution time Risk eg: Risk of viruses for a specific business unit Cost eg: Average cost of virus incident

Technical

eg: Percentage of systems with virus protection

Process

eg: Average virus incident resolution time

Risk

eg: Risk of viruses for a specific business unit

Cost

eg: Average cost of virus incident

Criteria for Good Metrics When determining the metrics that you will track, be SMART… S pecific M easurable A ttainable R epeatable T ime-dependent And… your results need to be ACTIONABLE !

When determining the metrics that you will track, be SMART…

S pecific

M easurable

A ttainable

R epeatable

T ime-dependent

And… your results need to be ACTIONABLE !

Metrics and Baselines Baselines are metrics that are compared to aggregated information about the past Common baselines Yesterday (“NASDAQ is up by a 100”) Average (“This took us 3 more time than the average case”) Maximum (“The traffic is spiking 300% over the maximum!”) Baselines enjoy the ease of interpretation ! What is the difference?

Baselines are metrics that are compared to aggregated information about the past

Common baselines

Yesterday (“NASDAQ is up by a 100”)

Average (“This took us 3 more time than the average case”)

Maximum (“The traffic is spiking 300% over the maximum!”)

Baselines enjoy the ease of interpretation !

Perceived Effectiveness of Metrics Source: CSO Magazine, 2005

Key Challenges with Security Metrics “Let’s make up some metrics…” Metrics must be based on policy to be effective Where is the data? Difficulties in acquiring raw data for measurement So many metrics, so little time… Which indicators should we compute and use? What does it mean? Interpreting the changes in measures parameters “Go back and try it again!!!” Challenges with presenting metrics to executives What do we do now? Defining actions and remediation planning

“Let’s make up some metrics…”

Metrics must be based on policy to be effective

Where is the data?

Difficulties in acquiring raw data for measurement

So many metrics, so little time…

Which indicators should we compute and use?

What does it mean?

Interpreting the changes in measures parameters

“Go back and try it again!!!”

Challenges with presenting metrics to executives

What do we do now?

Defining actions and remediation planning

Future of Security Metrics “ Information Security Management Metrics and Measurement” ( ISO 27004 ) in 2007-2009 Increased adoption of NIST 800-55 Legal and compliance drivers for standard metrics Growth of “ best practices ”-based metrics Convergence of security vendor metrics More automation of metrics Metrics and Crystal Ball Gazing Don’t Mix, but…

“ Information Security Management Metrics and Measurement” ( ISO 27004 ) in 2007-2009

Increased adoption of NIST 800-55

Legal and compliance drivers for standard metrics

Growth of “ best practices ”-based metrics

Convergence of security vendor metrics

More automation of metrics

Conclusion Security needs to be managed as a strategic business process (bla-bla-bla  ) Enterprise compliance and governance initiatives are driving security performance metrics Establishing a security metrics program is essential to determining if your security strategies are successful you are compliant In the future metrics will be based on standards

Security needs to be managed as a strategic business process (bla-bla-bla  )

Enterprise compliance and governance initiatives are driving security performance metrics

Establishing a security metrics program is essential to determining if

your security strategies are successful

you are compliant

In the future metrics will be based on standards

Thanks for Viewing Dr Anton Chuvakin http://www.chuvakin.org Also see my blog at www.securitywarrior.org

Dr Anton Chuvakin

http://www.chuvakin.org

Also see my blog at www.securitywarrior.org