Metrics, measures & Myths
•
16 likes•2,016 views
This document discusses metrics, measures, and myths related to security metrics. It begins by defining some key terms and presenting quotes emphasizing the importance of measurement. It then addresses five common myths about metrics and emphasizes that metrics should be used to identify opportunities and drive improvement, not to punish people. The document outlines characteristics of good metrics and provides examples of security metrics and key performance indicators. It discusses how metrics can be displayed in dashboards and how monitoring transforms data into useful security information and knowledge.
More Related Content
More from Ramsés Gallego
More from Ramsés Gallego (14)
Recently uploaded
Recently uploaded (20)
Metrics, measures & Myths
- 1. Metrics, Measures and Myths Ramsés Gallego CISM, CGEIT, CISSP, SCPM, ITIL, Six Sigma Black Belt Certified General Manager Entel Security & Risk Management rgallego@entel.es © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 2. Today’s agenda • Some quotes and definitions • The myths • The power of metrics • Metrics: characteristics & classification • What are CSFs, KGIs and KPIs? • Examples of security metrics and KPIs • SIM and MMI architectures • The SMART side of metrics © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 3. Let’s think about this © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 4. Let’s think about this • ‘Measure what is measurable and make measurable what is not so’ - Galileo Galilei (1564-1642) © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 5. Let’s think about this • ‘Measure what is measurable and make measurable what is not so’ - Galileo Galilei (1564-1642) • ‘If you cannot measure it, you cannot improve it’ - William Thomson (Lord Kelvin), (1824-1907) © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 6. Let’s think about this • ‘Measure what is measurable and make measurable what is not so’ - Galileo Galilei (1564-1642) • ‘If you cannot measure it, you cannot improve it’ - William Thomson (Lord Kelvin), (1824-1907) • ‘You cannot control what you cannot measure’ - DeMarco, 1982 © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 7. Let’s think about this • ‘Measure what is measurable and make measurable what is not so’ - Galileo Galilei (1564-1642) • ‘If you cannot measure it, you cannot improve it’ - William Thomson (Lord Kelvin), (1824-1907) • ‘You cannot control what you cannot measure’ - DeMarco, 1982 • ‘Even when it is not clear how we might measure an attribute, the act of proposing such measures will open a debate that leads to greater understanding’ - Fenton and Pfleeger, 1997 © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 8. Definitions © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 9. Definitions • Governance: “The set of responsibilities and practices exercised by the board and executive management with the goal of providing strategic direction, ensuring that objectives are achieved, ascertaining that risks are managed appropriately and veryfing that the enterprise’s resources are used responsibly” © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 10. Definitions: what is a metric? • The National Institute of Standards and Technology (NIST) define metrics as: ‘Tools designed to facilitate decision-making and improve performance and accountability through collection, analysis and reporting of relevant performance-related data’ • Metrics are simply a standard or system of measurement. In this case, it is a standard for measuring security, specifically measuring an organization’s security posture. Although there are some published standards for measuring security, ideally security metrics should be adjusted and tuned to fit a specific organization or situation © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 11. Goals of this effort • Develop a security metrics framework that allows management and operators to assess their security improvements (time-relevant), guide their security thinking and aid in risk assessment for their environments © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 12. Myths on metrics © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 13. Myths on metrics • #1 - a little data goes a long way © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 14. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 15. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 16. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 17. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas • #3 - we can’t measure what we cannot control © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 18. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas • #3 - we can’t measure what we cannot control – Fact: measure what you influence © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 19. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas • #3 - we can’t measure what we cannot control – Fact: measure what you influence • #4 - metrics are for measuring people © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 20. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas • #3 - we can’t measure what we cannot control – Fact: measure what you influence • #4 - metrics are for measuring people – Fact: measure the team contribution. They are an organizational tool © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 21. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas • #3 - we can’t measure what we cannot control – Fact: measure what you influence • #4 - metrics are for measuring people – Fact: measure the team contribution. They are an organizational tool • #5 - we must measure everything © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 22. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas • #3 - we can’t measure what we cannot control – Fact: measure what you influence • #4 - metrics are for measuring people – Fact: measure the team contribution. They are an organizational tool • #5 - we must measure everything – Fact: keep it simple so that everybody understands it © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 23. The power of metrics • It’s not in the details but in their clarity • Metrics allow executive management to: • Measure achievement • Drive performance • Improve and realign (towards goals) • Metrics should provide a holistic and balanced view of the business © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 24. Metrics: what is needed? • The 7 attributes of Information criteria (also known as the “IC Profile”) • Key conditions before defining a framework: • Having a pre-defined business process • Having clear goals/performance requirements • Having quantitative/qualitative measures for the business process © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 25. Metrics: Characteristics & Classification • Process Metrics • Objective/Subjective • Secure coding standards in use • Avg. time to correct critical vulnerabilities • Quantitative/Qualitative • Vulnerability metrics • Static/Dynamic • By vulnerability type • By ocurrence within a software development • Absolute/Relative life cycle phase • Management • Direct/Indirect • % of applications that are currently accepted by business partners • Trending: critical unresolved, accepted risks © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 26. Metric Specification © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 27. Metric Specification • Name of the metric • Description of what is measured • How is the metric measured • How often is the measurement taken • Range of values considered normal for the metric • Best possible value of the metric • Units of measurement © Source: Vicente Aceituno’s presentation for the FIST conferences in Madrid, 2008 © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 28. CSFs, KGIs and KPIs: what are they? • CSFs: Critical Success Factors or “vital elements” • KGIs: Key Goals Indicators or “what” has to be accomplished • KPIs: Key Performance Indicators or “how well” the process is performing © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 29. KGIs and KPIs reflect organizational goals © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 30. Example of IT metrics and KPIs • % reduction in repeat security incidents • Increased number of secure assets from risk analysis audits • % reduction of blank passwords on critical systems • % improvement on time-to-access applications • Improved bandwith use due to only-professional web surfing • % reduction in the unavailabilty of services and components (linked with corporate infrastructure management) • % efficiency improvement based on number of RFCs processed regarding vulnerabilities • % reduction in installed software not taken from DSL © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 31. Where do we show metrics?: Dahsboards and BSCs • Single point of information for infrastructure & security management • Help to make decisions and provide real-time answers to managers • Talk about the business, not about figures! • Need the involvement of the business and operations to be developed/designed in order to provide value • Web and role-based so as to get the right data (becoming the tool that consolidates siloed information) © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 32. Some dashboard examples © Business Objects. Crystal Xcelsius dashboard from www.xcelsius.com © Business Objects. Crystal Xcelsius dashboard from www.xcelsius.com © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 33. Some dashboard examples (II) © Business Objects. Crystal Xcelsius dashboard from www.xcelsius.com © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 34. Monitoring vs. Management T NG N E I M R GE TO NI A AN MO M Refine, analyze and Act on real business sort data that knowledge in a Value (and Cost) delivers security single place information according to Apply business Centralize access business need relevance to to data content information to and determine business applications priorities DATA INFORMATION ACTION KNOWLEDGE Level 2 Level 1 Level 3 Level 4 © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 35. The road to manage security information Alarm Escalation, Invoke Management Console, Response Response Model Management/Alert Management ● email ● Pager ● Cell ● ACTION Presentation Event Manage/Report Event Display, Trend Analysis, Security Reports, Performance Reports, Security System Health, Pattern Discovery KNOWLEDGE Assigning Ownership Prioritization Event Correlation Event Prioritization, Event Associations, Security Modeling Event Aggregation Log Data Reduction, Event Matching, Data Normalization and De-Duplicating Events Monitoring INFORMATION Reduction Data Filtering Event Monitoring, Third-Party Integration, Protocol DATA Data Repository Support Data Collection/Capture ● Syslog ● SNMP ● API ● © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 36. SIM and MMI Architectures query Policies Events Reporter Policy Manager Management Portal Collector ts er al Router Load Balancer SunOS Mainframe Windows X.500 Directory Router DB IDS Switch AIX Proxy Network Identity Applications /Hosts Security Systems Systems Information systems Systems © 2006 CA - All trademarks, trade names, services marks and logos referenced herein belong to their respective companies. © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 37. Using IT in the real world © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 38. Showing what really matters © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 39. Showing what really matters (II) © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 40. What can be achieved • KPIs that are a measure of how well a process is performing • The capability of predicting the probability of success or failure in the future • KPIs that are business-focused, process-oriented but IT-driven • KPIs that are expressed in precisely measurable terms • KPIs that, when acted upon, will help to improve the process • FOCUS on what is really important and has impact © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 41. The SMART side of metrics First business needs, then processes, then metrics, • then tools • Keep them simple • Use “as is/to be” & “is/is not” lists • Metrics should be S-M-A-R-T © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
- 42. THANK YOU Metrics, Measures and Myths Ramsés Gallego CISM, CGEIT, CISSP, SCPM, ITIL, Six Sigma Black Belt Certified General Manager Entel Security & Risk Management rgallego@entel.es © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
Editor's Notes
- Let’s have a look to today’s main points in the agenda. First of all we are going to see the power of metrics and how important they are to know what is happening in a company and how the enterprise is doing regarding bottom-line impact. Metrics are the indicators that tell not only management but also people on day-to-day operations how well they are performing to already established goals and business objectives. As we will see later, there is way (and a deep need, in my opinion) to align security management with the business. We will also make a quick overview of what are CSFs, KGIs and KPIs and the intimate relationship between them. As a security practitioner and consultant, I will give you some real examples of KPIs and how they integrate in a balanced scorecard and also talk about a real implementation of a security dashboard on a customer. Finally, to wrap up, we will see the SMART side of metrics and a quick summary. Let’s go
- Objectives need to be defined The course in charted Risks are identified, evaluated and managed Resources and their criticality and sensitivity are determined Objectives are: Strategic alignment Risk Management Business process assurance Value delivery Resource Management Performance measurement
- It is said that you cannot manage what you cannot measure (and I fully agree with that vision) and my colleague Krag Brotby will later on the day do a presentation about it. It has to be pointed out that normalization of data it is very useful since you have to be able to compare between departments and divisions but also with other industry peers. Normalization places all the measures on a similar footing by equalizing them across a common organizational base Besides, metrics are rarely raw data but some derivative number (ratio, index, percentage or weighted average) Critical to successful implementation of metrics is the understanding and acceptance that they take an important commitment and use in time and resources
- Regarding IC, each organization needs to decide how important each attribute is for their business and this profile expresses the enterprise’s position and appetite for risk
- CSFs were introduced by John F. Rockart in 1979 and are defined as elements that are vital for a strategy to be successful. In another level they could also be seen as important things for the process in this way: “what you need from others” and “what you can do yourself and deliver to others” KGIs are a target to achieve, a measure of outcome We are going to focus today in KPIs since they are the day-to-day metrics, the one being monitored constantly In this context we need to remember that IT is a major enabler of the business and, therefore, KPIs are a measure of performance As you can see in the graphic on the left, KGIs are just above generic IT goals and KPIs are next to IT processes showing their area or influence. Consequently, we could define KGIs as “lag” indicators while KPIs could be “lead” indicators. By the way, both measures could also be expressed negatively showing not having reached the goal or not performing well KPIs have a cause-effect relationship with KGIs of the process In summary, KGIs are business-driven while KPIs are process-oriented
- I think that KGIs and KPIs do reflect organizational goals. Once a company has analyzed its mission, identified all its stakeholders and defined its goals, it needs a way to measure progress. KPIs are those measurements. Take into account that some analysts and consultants call KPIs also KSI (Key Success Indicators) but it is extremely more common the former acronym (with a P from performance) giving it a sense of direction and continuous monitoring. Top-down approach KPIs are quantifiable measurements, agreed to beforehand. However, I would like to deviate from the idea that there is a kind of negotiation with KGIs and KPIs. There should be an agreement but what really matters is the strategy and how a company is going to measure the achievement of the target. In the same way, scaling down to the IT or security department, there should be an agreement (again, not a biased negotiation) of what is needed and how security brings and adds value to the business (by preventing threats exploiting a vulnerability better than last month or year or some other measures that we are going to see in a moment).
- This takes us to a whole new level of data visualization and integration: dashboards and balanced scorecards. Introduced by Robert Kaplan and David Norton in the early 90s, (1992 to be precise), balanced scorecards convert strategy into action by showing in a centralized single place all the metrics that executive management needs to take decisions. In fact, not only management but also operational teams and divisional managers are empowered by balanced scorecards since different views and information is provided depending on the role and profile of the viewer. The definition of BSCs given by Mr. Kaplan and Mr. Norton is very interesting. Listen for the words: comprehensive view, performance, management tool. A BSC is a method and a management tool for ensuring enterprise’s activities in terms of its vision and strategies by giving managers a fast, comprehensive view of the performance of a business. It is here where we should introduce the 4 different perspectives of a balanced scorecard: financial, customer, internal process and learning/innovation. Scorecards - Most strategic level of the business decision while dashboard work more in the operational side giving key users metrics of their area of influence
- Level 0 - Non existent Level 1 - Initial Level 2 - Repeatable Level 3 - Defined Level 4 - Managed Level 5 - Optimised “Knowledge resides in the person, not in the data…it is the response and action to information that counts”
- We built upon other disciplines like network management, asset management (CMDB) and storage management (backup & contigency plan) so as to provide a unique repository of information and began escalating in what we called “The road to management” “You need to know what you have to be able to protect it”
- 3-layer architecture
- We focused so much in showing a KPI regarding critical operations, which nodes out of 1453 where at risk and, consequently, which operations were being threatened. Remember, at this point the definition of what is risk: the potential that a given threat will exploit a vulnerability with an impact in an asset or group of assets
- (meaning alignment with the business) (since KPIs are “lead” indicators) FOCUS
- SIMPLE MEASURABLE ACHIEVEABLE REALISTIC TIME-DRIVEN