SlideShare a Scribd company logo
Metrics, Measures and Myths

                                                                          Ramsés Gallego
                                                    CISM, CGEIT, CISSP, SCPM, ITIL, Six Sigma Black Belt Certified
                                                                         General Manager
                                                                 Entel Security & Risk Management
                                                                         rgallego@entel.es



                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Today’s agenda

                            • Some quotes and definitions
                            • The myths
                            • The power of metrics
                            • Metrics: characteristics & classification
                            • What are CSFs, KGIs and KPIs?
                            • Examples of security metrics and KPIs
                            • SIM and MMI architectures
                            • The SMART side of metrics


                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Let’s think about this




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Let’s think about this
                   • ‘Measure what is measurable and make measurable what is not so’
                        - Galileo Galilei (1564-1642)




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Let’s think about this
                   • ‘Measure what is measurable and make measurable what is not so’
                        - Galileo Galilei (1564-1642)

                   • ‘If you cannot measure it, you cannot improve it’ - William Thomson
                        (Lord Kelvin), (1824-1907)




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Let’s think about this
                   • ‘Measure what is measurable and make measurable what is not so’
                        - Galileo Galilei (1564-1642)

                   • ‘If you cannot measure it, you cannot improve it’ - William Thomson
                        (Lord Kelvin), (1824-1907)

                   • ‘You cannot control what you cannot measure’ - DeMarco, 1982




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Let’s think about this
                   • ‘Measure what is measurable and make measurable what is not so’
                        - Galileo Galilei (1564-1642)

                   • ‘If you cannot measure it, you cannot improve it’ - William Thomson
                        (Lord Kelvin), (1824-1907)

                   • ‘You cannot control what you cannot measure’ - DeMarco, 1982
                   • ‘Even when it is not clear how we might measure an attribute, the act
                        of proposing such measures will open a debate that leads to greater
                        understanding’ - Fenton and Pfleeger, 1997

                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Definitions




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Definitions

                        • Governance: “The set of responsibilities and practices

                        exercised by the board and executive management with the

                        goal of providing strategic direction, ensuring that

                        objectives are achieved, ascertaining that risks are

                        managed appropriately and veryfing that the enterprise’s

                        resources are used responsibly”




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Definitions: what is a metric?
                       • The National Institute of Standards and Technology (NIST) define

                            metrics as: ‘Tools designed to facilitate decision-making and

                            improve performance and accountability through collection,

                            analysis and reporting of relevant performance-related data’

                       • Metrics are simply a standard or system of measurement. In this case,
                            it is a standard for measuring security, specifically measuring an
                            organization’s security posture. Although there are some published
                            standards for measuring security, ideally security metrics should be
                            adjusted and tuned to fit a specific organization or situation



                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Goals of this effort
                        • Develop a security metrics framework

                        that allows management and operators

                        to assess their security improvements

                        (time-relevant), guide their security

                        thinking and aid in risk assessment for

                        their environments


                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Myths on metrics




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Myths on metrics
                       • #1 - a little data goes a long way




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Myths on metrics
                       • #1 - a little data goes a long way
                            – Fact: you can only improve what you measure




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Myths on metrics
                       • #1 - a little data goes a long way
                            – Fact: you can only improve what you measure

                       • #2 - measurement is for punishing the guilty




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Myths on metrics
                       • #1 - a little data goes a long way
                            – Fact: you can only improve what you measure

                       • #2 - measurement is for punishing the guilty
                            – Fact: metrics are for problem solving and identifying opportunity areas




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Myths on metrics
                       • #1 - a little data goes a long way
                            – Fact: you can only improve what you measure

                       • #2 - measurement is for punishing the guilty
                            – Fact: metrics are for problem solving and identifying opportunity areas

                       • #3 - we can’t measure what we cannot control




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Myths on metrics
                       • #1 - a little data goes a long way
                            – Fact: you can only improve what you measure

                       • #2 - measurement is for punishing the guilty
                            – Fact: metrics are for problem solving and identifying opportunity areas

                       • #3 - we can’t measure what we cannot control
                            – Fact: measure what you influence




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Myths on metrics
                       • #1 - a little data goes a long way
                            – Fact: you can only improve what you measure

                       • #2 - measurement is for punishing the guilty
                            – Fact: metrics are for problem solving and identifying opportunity areas

                       • #3 - we can’t measure what we cannot control
                            – Fact: measure what you influence

                       • #4 - metrics are for measuring people




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Myths on metrics
                       • #1 - a little data goes a long way
                            – Fact: you can only improve what you measure

                       • #2 - measurement is for punishing the guilty
                            – Fact: metrics are for problem solving and identifying opportunity areas

                       • #3 - we can’t measure what we cannot control
                            – Fact: measure what you influence

                       • #4 - metrics are for measuring people
                            – Fact: measure the team contribution. They are an organizational tool




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Myths on metrics
                       • #1 - a little data goes a long way
                            – Fact: you can only improve what you measure

                       • #2 - measurement is for punishing the guilty
                            – Fact: metrics are for problem solving and identifying opportunity areas

                       • #3 - we can’t measure what we cannot control
                            – Fact: measure what you influence

                       • #4 - metrics are for measuring people
                            – Fact: measure the team contribution. They are an organizational tool

                       • #5 - we must measure everything




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Myths on metrics
                       • #1 - a little data goes a long way
                            – Fact: you can only improve what you measure

                       • #2 - measurement is for punishing the guilty
                            – Fact: metrics are for problem solving and identifying opportunity areas

                       • #3 - we can’t measure what we cannot control
                            – Fact: measure what you influence

                       • #4 - metrics are for measuring people
                            – Fact: measure the team contribution. They are an organizational tool

                       • #5 - we must measure everything
                            – Fact: keep it simple so that everybody understands it


                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
The power of metrics
                            • It’s not in the details but in their clarity

                            • Metrics allow executive management to:

                              • Measure achievement

                              • Drive performance

                              • Improve and realign (towards goals)

                            • Metrics should provide a holistic and balanced view of

                            the business


                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Metrics: what is needed?
                            • The 7 attributes of Information criteria (also known as the “IC

                            Profile”)


                                                      • Key conditions before defining a framework:
                                                        • Having a pre-defined business process

                                                        • Having clear goals/performance requirements

                                                        • Having quantitative/qualitative measures for the

                                                        business process




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Metrics: Characteristics & Classification
                                                                 • Process Metrics

               • Objective/Subjective                             • Secure coding standards in use

                                                                  • Avg. time to correct critical vulnerabilities

               • Quantitative/Qualitative                        • Vulnerability metrics
               • Static/Dynamic                                    • By vulnerability type

                                                                   • By ocurrence within a software development

               • Absolute/Relative                                 life cycle phase

                                                                 • Management
               • Direct/Indirect
                                                                   • % of applications that are currently accepted

                                                                   by business partners

                                                                   • Trending: critical unresolved, accepted risks



                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Metric Specification




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Metric Specification

                      • Name of the metric
                      • Description of what is measured
                      • How is the metric measured
                      • How often is the measurement taken
                      • Range of values considered normal for the metric
                      • Best possible value of the metric
                      • Units of measurement

                       © Source: Vicente Aceituno’s presentation for the FIST conferences in Madrid, 2008




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
CSFs, KGIs and KPIs: what are they?

                                                               • CSFs: Critical Success Factors or
                                                               “vital elements”

                                                               • KGIs: Key Goals Indicators or

                                                               “what” has to be accomplished

                                                               • KPIs: Key Performance Indicators

                                                               or “how well” the process is

                                                               performing




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
KGIs and KPIs reflect organizational goals




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Example of IT metrics and KPIs
                            • % reduction in repeat security incidents
                            • Increased number of secure assets from risk analysis audits
                            • % reduction of blank passwords on critical systems
                            • % improvement on time-to-access applications
                            • Improved bandwith use due to only-professional web surfing
                            • % reduction in the unavailabilty of services and components (linked with
                            corporate infrastructure management)
                            • % efficiency improvement based on number of RFCs processed regarding
                            vulnerabilities
                            • % reduction in installed software not taken from DSL




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Where do we show metrics?:
                                                                            Dahsboards and BSCs

                       • Single point of information for infrastructure & security management

                       • Help to make decisions and provide real-time answers to managers

                       • Talk about the business, not about figures!

                       • Need the involvement of the business and operations to be

                       developed/designed in order to provide value

                       • Web and role-based so as to get the right data (becoming the tool

                       that consolidates siloed information)

                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Some dashboard examples




                                                                                                                                     © Business Objects. Crystal Xcelsius dashboard from www.xcelsius.com




                                                    © Business Objects. Crystal Xcelsius dashboard from www.xcelsius.com



                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Some dashboard examples (II)




                                                    © Business Objects. Crystal Xcelsius dashboard from www.xcelsius.com




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Monitoring vs. Management


                                                                                                                           T
                                                                         NG                                              N
                                                                                                                        E
                                                                        I                                             M
                                                                       R
                                                                                                                   GE
                                                                    TO
                                                                 NI                                              A
                                                                                                           AN
                                                          MO                                             M
                                                                           Refine, analyze and                          Act on real business
                                                                              sort data that                            knowledge in a
                               Value (and Cost)




                                                                            delivers security                           single place
                                                                               information                              according to
                                                                                                 Apply business
                                                  Centralize access                                                     business need
                                                                                                 relevance to
                                                   to data content
                                                                                                 information to
                                                         and
                                                                                                 determine business
                                                     applications
                                                                                                 priorities


                                                            DATA             INFORMATION                                       ACTION
                                                                                                   KNOWLEDGE

                                                                                 Level 2
                                                       Level 1                                         Level 3                 Level 4




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
The road to manage security information

                                                                                          Alarm Escalation, Invoke Management Console,
                                                                    Response
                                                                                          Response Model
                                                                 Management/Alert
                             Management

                                                                                          ● email ● Pager ● Cell ●
                                                      ACTION
                                                                   Presentation
                                                                Event Manage/Report
                                                                                          Event Display, Trend Analysis, Security Reports,
                                                                                          Performance Reports, Security System Health,
                                                                  Pattern Discovery
                                                    KNOWLEDGE                             Assigning Ownership
                                                                    Prioritization


                                                                  Event Correlation
                                                                                          Event Prioritization, Event Associations,
                                                                                          Security Modeling
                                                                 Event Aggregation

                                                                                          Log Data Reduction, Event Matching,
                                                                Data Normalization and    De-Duplicating Events
                             Monitoring




                                              INFORMATION             Reduction

                                                                    Data Filtering

                                                                                          Event Monitoring, Third-Party Integration, Protocol
                                                         DATA      Data Repository        Support
                                                                Data Collection/Capture   ● Syslog ● SNMP ● API ●


                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
SIM and MMI Architectures


                                                                                                          query

                            Policies                     Events                                                                              Reporter
                                                                                                         Policy
                                                                                                         Manager


                                                                                                                                                                                                  Management Portal
                                                                                 Collector




                                                                                                                       ts
                                                                                                                     er
                                                                                                                   al
                                                                                                                                                                                                      Router

                             Load Balancer                                                             SunOS                          Mainframe               Windows
                                                         X.500 Directory
                                                Router                      DB                                                                                                                                     IDS
                                                Switch                                                                    AIX
                                                                                                                                                                                                                         Proxy
                                             Network             Identity                                     Applications /Hosts                                                                       Security
                                             Systems             Systems                                      Information systems                                                                       Systems


                                                                                 © 2006 CA - All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Using IT in the real world




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Showing what really matters




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
Showing what really matters (II)




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
What can be achieved


                            • KPIs that are a measure of how well a process is performing
                            • The capability of predicting the probability of success or failure in the
                            future
                            • KPIs that are business-focused, process-oriented but IT-driven
                            • KPIs that are expressed in precisely measurable terms
                            • KPIs that, when acted upon, will help to improve the process
                            • FOCUS on what is really important and has impact




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
The SMART side of metrics




                            First business needs, then processes, then metrics,
                       •

                       then tools
                       • Keep them simple
                       • Use “as is/to be” & “is/is not” lists
                       • Metrics should be S-M-A-R-T




                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009
THANK YOU
                                                        Metrics, Measures and Myths
                                                                          Ramsés Gallego
                                                    CISM, CGEIT, CISSP, SCPM, ITIL, Six Sigma Black Belt Certified
                                                                         General Manager
                                                                 Entel Security & Risk Management
                                                                         rgallego@entel.es



                © 2008 ISACA. All rights reserved

Wednesday, March 25, 2009

More Related Content

More from Ramsés Gallego

IT Controls Cloud Webinar - ISACA
IT Controls Cloud Webinar - ISACAIT Controls Cloud Webinar - ISACA
IT Controls Cloud Webinar - ISACA
Ramsés Gallego
 
The Perfect Storm
The Perfect StormThe Perfect Storm
The Perfect Storm
Ramsés Gallego
 
ISACA Barcelona Chapter Congress - July 2011
ISACA Barcelona Chapter Congress - July 2011ISACA Barcelona Chapter Congress - July 2011
ISACA Barcelona Chapter Congress - July 2011
Ramsés Gallego
 
Culture structure strategy_for_a_grc_program
Culture structure strategy_for_a_grc_programCulture structure strategy_for_a_grc_program
Culture structure strategy_for_a_grc_program
Ramsés Gallego
 
Strategic governance performance_management_systems
Strategic governance performance_management_systemsStrategic governance performance_management_systems
Strategic governance performance_management_systems
Ramsés Gallego
 
Modern cyber threats_and_how_to_combat_them_panel
Modern cyber threats_and_how_to_combat_them_panelModern cyber threats_and_how_to_combat_them_panel
Modern cyber threats_and_how_to_combat_them_panel
Ramsés Gallego
 
From technology risk_to_enterprise_risk_the_new_frontier
From technology risk_to_enterprise_risk_the_new_frontierFrom technology risk_to_enterprise_risk_the_new_frontier
From technology risk_to_enterprise_risk_the_new_frontier
Ramsés Gallego
 
Entel Service Management
Entel Service ManagementEntel Service Management
Entel Service Management
Ramsés Gallego
 
Malware mitigation
Malware mitigationMalware mitigation
Malware mitigation
Ramsés Gallego
 
DLP - Network Security Conference_ Ramsés Gallego
DLP - Network Security Conference_ Ramsés GallegoDLP - Network Security Conference_ Ramsés Gallego
DLP - Network Security Conference_ Ramsés Gallego
Ramsés Gallego
 
e-Symposium_ISACA_Ramsés_Gallego
e-Symposium_ISACA_Ramsés_Gallegoe-Symposium_ISACA_Ramsés_Gallego
e-Symposium_ISACA_Ramsés_Gallego
Ramsés Gallego
 
Entel SSO
Entel SSOEntel SSO
Entel SSO
Ramsés Gallego
 
Entel DLP
Entel DLPEntel DLP
Entel DLP
Ramsés Gallego
 
Entel S&RM
Entel S&RMEntel S&RM
Entel S&RM
Ramsés Gallego
 

More from Ramsés Gallego (14)

IT Controls Cloud Webinar - ISACA
IT Controls Cloud Webinar - ISACAIT Controls Cloud Webinar - ISACA
IT Controls Cloud Webinar - ISACA
 
The Perfect Storm
The Perfect StormThe Perfect Storm
The Perfect Storm
 
ISACA Barcelona Chapter Congress - July 2011
ISACA Barcelona Chapter Congress - July 2011ISACA Barcelona Chapter Congress - July 2011
ISACA Barcelona Chapter Congress - July 2011
 
Culture structure strategy_for_a_grc_program
Culture structure strategy_for_a_grc_programCulture structure strategy_for_a_grc_program
Culture structure strategy_for_a_grc_program
 
Strategic governance performance_management_systems
Strategic governance performance_management_systemsStrategic governance performance_management_systems
Strategic governance performance_management_systems
 
Modern cyber threats_and_how_to_combat_them_panel
Modern cyber threats_and_how_to_combat_them_panelModern cyber threats_and_how_to_combat_them_panel
Modern cyber threats_and_how_to_combat_them_panel
 
From technology risk_to_enterprise_risk_the_new_frontier
From technology risk_to_enterprise_risk_the_new_frontierFrom technology risk_to_enterprise_risk_the_new_frontier
From technology risk_to_enterprise_risk_the_new_frontier
 
Entel Service Management
Entel Service ManagementEntel Service Management
Entel Service Management
 
Malware mitigation
Malware mitigationMalware mitigation
Malware mitigation
 
DLP - Network Security Conference_ Ramsés Gallego
DLP - Network Security Conference_ Ramsés GallegoDLP - Network Security Conference_ Ramsés Gallego
DLP - Network Security Conference_ Ramsés Gallego
 
e-Symposium_ISACA_Ramsés_Gallego
e-Symposium_ISACA_Ramsés_Gallegoe-Symposium_ISACA_Ramsés_Gallego
e-Symposium_ISACA_Ramsés_Gallego
 
Entel SSO
Entel SSOEntel SSO
Entel SSO
 
Entel DLP
Entel DLPEntel DLP
Entel DLP
 
Entel S&RM
Entel S&RMEntel S&RM
Entel S&RM
 

Recently uploaded

How are Lilac French Bulldogs Beauty Charming the World and Capturing Hearts....
How are Lilac French Bulldogs Beauty Charming the World and Capturing Hearts....How are Lilac French Bulldogs Beauty Charming the World and Capturing Hearts....
How are Lilac French Bulldogs Beauty Charming the World and Capturing Hearts....
Lacey Max
 
Best Competitive Marble Pricing in Dubai - ☎ 9928909666
Best Competitive Marble Pricing in Dubai - ☎ 9928909666Best Competitive Marble Pricing in Dubai - ☎ 9928909666
Best Competitive Marble Pricing in Dubai - ☎ 9928909666
Stone Art Hub
 
The 10 Most Influential Leaders Guiding Corporate Evolution, 2024.pdf
The 10 Most Influential Leaders Guiding Corporate Evolution, 2024.pdfThe 10 Most Influential Leaders Guiding Corporate Evolution, 2024.pdf
The 10 Most Influential Leaders Guiding Corporate Evolution, 2024.pdf
thesiliconleaders
 
Digital Marketing with a Focus on Sustainability
Digital Marketing with a Focus on SustainabilityDigital Marketing with a Focus on Sustainability
Digital Marketing with a Focus on Sustainability
sssourabhsharma
 
Pitch Deck Teardown: Kinnect's $250k Angel deck
Pitch Deck Teardown: Kinnect's $250k Angel deckPitch Deck Teardown: Kinnect's $250k Angel deck
Pitch Deck Teardown: Kinnect's $250k Angel deck
HajeJanKamps
 
❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka Result Satta Matka Guessing Satta Fix jodi Kalyan Fin...
❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka Result Satta Matka Guessing Satta Fix jodi Kalyan Fin...❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka Result Satta Matka Guessing Satta Fix jodi Kalyan Fin...
❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka Result Satta Matka Guessing Satta Fix jodi Kalyan Fin...
❼❷⓿❺❻❷❽❷❼❽ Dpboss Kalyan Satta Matka Guessing Matka Result Main Bazar chart
 
Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...
Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...
Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...
bosssp10
 
Authentically Social by Corey Perlman - EO Puerto Rico
Authentically Social by Corey Perlman - EO Puerto RicoAuthentically Social by Corey Perlman - EO Puerto Rico
Authentically Social by Corey Perlman - EO Puerto Rico
Corey Perlman, Social Media Speaker and Consultant
 
Anny Serafina Love - Letter of Recommendation by Kellen Harkins, MS.
Anny Serafina Love - Letter of Recommendation by Kellen Harkins, MS.Anny Serafina Love - Letter of Recommendation by Kellen Harkins, MS.
Anny Serafina Love - Letter of Recommendation by Kellen Harkins, MS.
AnnySerafinaLove
 
Zodiac Signs and Food Preferences_ What Your Sign Says About Your Taste
Zodiac Signs and Food Preferences_ What Your Sign Says About Your TasteZodiac Signs and Food Preferences_ What Your Sign Says About Your Taste
Zodiac Signs and Food Preferences_ What Your Sign Says About Your Taste
my Pandit
 
DearbornMusic-KatherineJasperFullSailUni
DearbornMusic-KatherineJasperFullSailUniDearbornMusic-KatherineJasperFullSailUni
DearbornMusic-KatherineJasperFullSailUni
katiejasper96
 
3 Simple Steps To Buy Verified Payoneer Account In 2024
3 Simple Steps To Buy Verified Payoneer Account In 20243 Simple Steps To Buy Verified Payoneer Account In 2024
3 Simple Steps To Buy Verified Payoneer Account In 2024
SEOSMMEARTH
 
How to Implement a Real Estate CRM Software
How to Implement a Real Estate CRM SoftwareHow to Implement a Real Estate CRM Software
How to Implement a Real Estate CRM Software
SalesTown
 
Building Your Employer Brand with Social Media
Building Your Employer Brand with Social MediaBuilding Your Employer Brand with Social Media
Building Your Employer Brand with Social Media
LuanWise
 
Income Tax exemption for Start up : Section 80 IAC
Income Tax  exemption for Start up : Section 80 IACIncome Tax  exemption for Start up : Section 80 IAC
Income Tax exemption for Start up : Section 80 IAC
CA Dr. Prithvi Ranjan Parhi
 
Innovation Management Frameworks: Your Guide to Creativity & Innovation
Innovation Management Frameworks: Your Guide to Creativity & InnovationInnovation Management Frameworks: Your Guide to Creativity & Innovation
Innovation Management Frameworks: Your Guide to Creativity & Innovation
Operational Excellence Consulting
 
Unveiling the Dynamic Personalities, Key Dates, and Horoscope Insights: Gemin...
Unveiling the Dynamic Personalities, Key Dates, and Horoscope Insights: Gemin...Unveiling the Dynamic Personalities, Key Dates, and Horoscope Insights: Gemin...
Unveiling the Dynamic Personalities, Key Dates, and Horoscope Insights: Gemin...
my Pandit
 
Maksym Vyshnivetskyi: PMO KPIs (UA) (#12)
Maksym Vyshnivetskyi: PMO KPIs (UA) (#12)Maksym Vyshnivetskyi: PMO KPIs (UA) (#12)
Maksym Vyshnivetskyi: PMO KPIs (UA) (#12)
Lviv Startup Club
 
The Genesis of BriansClub.cm Famous Dark WEb Platform
The Genesis of BriansClub.cm Famous Dark WEb PlatformThe Genesis of BriansClub.cm Famous Dark WEb Platform
The Genesis of BriansClub.cm Famous Dark WEb Platform
SabaaSudozai
 
Understanding User Needs and Satisfying Them
Understanding User Needs and Satisfying ThemUnderstanding User Needs and Satisfying Them
Understanding User Needs and Satisfying Them
Aggregage
 

Recently uploaded (20)

How are Lilac French Bulldogs Beauty Charming the World and Capturing Hearts....
How are Lilac French Bulldogs Beauty Charming the World and Capturing Hearts....How are Lilac French Bulldogs Beauty Charming the World and Capturing Hearts....
How are Lilac French Bulldogs Beauty Charming the World and Capturing Hearts....
 
Best Competitive Marble Pricing in Dubai - ☎ 9928909666
Best Competitive Marble Pricing in Dubai - ☎ 9928909666Best Competitive Marble Pricing in Dubai - ☎ 9928909666
Best Competitive Marble Pricing in Dubai - ☎ 9928909666
 
The 10 Most Influential Leaders Guiding Corporate Evolution, 2024.pdf
The 10 Most Influential Leaders Guiding Corporate Evolution, 2024.pdfThe 10 Most Influential Leaders Guiding Corporate Evolution, 2024.pdf
The 10 Most Influential Leaders Guiding Corporate Evolution, 2024.pdf
 
Digital Marketing with a Focus on Sustainability
Digital Marketing with a Focus on SustainabilityDigital Marketing with a Focus on Sustainability
Digital Marketing with a Focus on Sustainability
 
Pitch Deck Teardown: Kinnect's $250k Angel deck
Pitch Deck Teardown: Kinnect's $250k Angel deckPitch Deck Teardown: Kinnect's $250k Angel deck
Pitch Deck Teardown: Kinnect's $250k Angel deck
 
❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka Result Satta Matka Guessing Satta Fix jodi Kalyan Fin...
❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka Result Satta Matka Guessing Satta Fix jodi Kalyan Fin...❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka Result Satta Matka Guessing Satta Fix jodi Kalyan Fin...
❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka Result Satta Matka Guessing Satta Fix jodi Kalyan Fin...
 
Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...
Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...
Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...
 
Authentically Social by Corey Perlman - EO Puerto Rico
Authentically Social by Corey Perlman - EO Puerto RicoAuthentically Social by Corey Perlman - EO Puerto Rico
Authentically Social by Corey Perlman - EO Puerto Rico
 
Anny Serafina Love - Letter of Recommendation by Kellen Harkins, MS.
Anny Serafina Love - Letter of Recommendation by Kellen Harkins, MS.Anny Serafina Love - Letter of Recommendation by Kellen Harkins, MS.
Anny Serafina Love - Letter of Recommendation by Kellen Harkins, MS.
 
Zodiac Signs and Food Preferences_ What Your Sign Says About Your Taste
Zodiac Signs and Food Preferences_ What Your Sign Says About Your TasteZodiac Signs and Food Preferences_ What Your Sign Says About Your Taste
Zodiac Signs and Food Preferences_ What Your Sign Says About Your Taste
 
DearbornMusic-KatherineJasperFullSailUni
DearbornMusic-KatherineJasperFullSailUniDearbornMusic-KatherineJasperFullSailUni
DearbornMusic-KatherineJasperFullSailUni
 
3 Simple Steps To Buy Verified Payoneer Account In 2024
3 Simple Steps To Buy Verified Payoneer Account In 20243 Simple Steps To Buy Verified Payoneer Account In 2024
3 Simple Steps To Buy Verified Payoneer Account In 2024
 
How to Implement a Real Estate CRM Software
How to Implement a Real Estate CRM SoftwareHow to Implement a Real Estate CRM Software
How to Implement a Real Estate CRM Software
 
Building Your Employer Brand with Social Media
Building Your Employer Brand with Social MediaBuilding Your Employer Brand with Social Media
Building Your Employer Brand with Social Media
 
Income Tax exemption for Start up : Section 80 IAC
Income Tax  exemption for Start up : Section 80 IACIncome Tax  exemption for Start up : Section 80 IAC
Income Tax exemption for Start up : Section 80 IAC
 
Innovation Management Frameworks: Your Guide to Creativity & Innovation
Innovation Management Frameworks: Your Guide to Creativity & InnovationInnovation Management Frameworks: Your Guide to Creativity & Innovation
Innovation Management Frameworks: Your Guide to Creativity & Innovation
 
Unveiling the Dynamic Personalities, Key Dates, and Horoscope Insights: Gemin...
Unveiling the Dynamic Personalities, Key Dates, and Horoscope Insights: Gemin...Unveiling the Dynamic Personalities, Key Dates, and Horoscope Insights: Gemin...
Unveiling the Dynamic Personalities, Key Dates, and Horoscope Insights: Gemin...
 
Maksym Vyshnivetskyi: PMO KPIs (UA) (#12)
Maksym Vyshnivetskyi: PMO KPIs (UA) (#12)Maksym Vyshnivetskyi: PMO KPIs (UA) (#12)
Maksym Vyshnivetskyi: PMO KPIs (UA) (#12)
 
The Genesis of BriansClub.cm Famous Dark WEb Platform
The Genesis of BriansClub.cm Famous Dark WEb PlatformThe Genesis of BriansClub.cm Famous Dark WEb Platform
The Genesis of BriansClub.cm Famous Dark WEb Platform
 
Understanding User Needs and Satisfying Them
Understanding User Needs and Satisfying ThemUnderstanding User Needs and Satisfying Them
Understanding User Needs and Satisfying Them
 

Metrics, measures & Myths

  • 1. Metrics, Measures and Myths Ramsés Gallego CISM, CGEIT, CISSP, SCPM, ITIL, Six Sigma Black Belt Certified General Manager Entel Security & Risk Management rgallego@entel.es © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 2. Today’s agenda • Some quotes and definitions • The myths • The power of metrics • Metrics: characteristics & classification • What are CSFs, KGIs and KPIs? • Examples of security metrics and KPIs • SIM and MMI architectures • The SMART side of metrics © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 3. Let’s think about this © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 4. Let’s think about this • ‘Measure what is measurable and make measurable what is not so’ - Galileo Galilei (1564-1642) © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 5. Let’s think about this • ‘Measure what is measurable and make measurable what is not so’ - Galileo Galilei (1564-1642) • ‘If you cannot measure it, you cannot improve it’ - William Thomson (Lord Kelvin), (1824-1907) © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 6. Let’s think about this • ‘Measure what is measurable and make measurable what is not so’ - Galileo Galilei (1564-1642) • ‘If you cannot measure it, you cannot improve it’ - William Thomson (Lord Kelvin), (1824-1907) • ‘You cannot control what you cannot measure’ - DeMarco, 1982 © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 7. Let’s think about this • ‘Measure what is measurable and make measurable what is not so’ - Galileo Galilei (1564-1642) • ‘If you cannot measure it, you cannot improve it’ - William Thomson (Lord Kelvin), (1824-1907) • ‘You cannot control what you cannot measure’ - DeMarco, 1982 • ‘Even when it is not clear how we might measure an attribute, the act of proposing such measures will open a debate that leads to greater understanding’ - Fenton and Pfleeger, 1997 © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 8. Definitions © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 9. Definitions • Governance: “The set of responsibilities and practices exercised by the board and executive management with the goal of providing strategic direction, ensuring that objectives are achieved, ascertaining that risks are managed appropriately and veryfing that the enterprise’s resources are used responsibly” © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 10. Definitions: what is a metric? • The National Institute of Standards and Technology (NIST) define metrics as: ‘Tools designed to facilitate decision-making and improve performance and accountability through collection, analysis and reporting of relevant performance-related data’ • Metrics are simply a standard or system of measurement. In this case, it is a standard for measuring security, specifically measuring an organization’s security posture. Although there are some published standards for measuring security, ideally security metrics should be adjusted and tuned to fit a specific organization or situation © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 11. Goals of this effort • Develop a security metrics framework that allows management and operators to assess their security improvements (time-relevant), guide their security thinking and aid in risk assessment for their environments © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 12. Myths on metrics © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 13. Myths on metrics • #1 - a little data goes a long way © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 14. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 15. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 16. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 17. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas • #3 - we can’t measure what we cannot control © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 18. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas • #3 - we can’t measure what we cannot control – Fact: measure what you influence © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 19. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas • #3 - we can’t measure what we cannot control – Fact: measure what you influence • #4 - metrics are for measuring people © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 20. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas • #3 - we can’t measure what we cannot control – Fact: measure what you influence • #4 - metrics are for measuring people – Fact: measure the team contribution. They are an organizational tool © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 21. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas • #3 - we can’t measure what we cannot control – Fact: measure what you influence • #4 - metrics are for measuring people – Fact: measure the team contribution. They are an organizational tool • #5 - we must measure everything © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 22. Myths on metrics • #1 - a little data goes a long way – Fact: you can only improve what you measure • #2 - measurement is for punishing the guilty – Fact: metrics are for problem solving and identifying opportunity areas • #3 - we can’t measure what we cannot control – Fact: measure what you influence • #4 - metrics are for measuring people – Fact: measure the team contribution. They are an organizational tool • #5 - we must measure everything – Fact: keep it simple so that everybody understands it © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 23. The power of metrics • It’s not in the details but in their clarity • Metrics allow executive management to: • Measure achievement • Drive performance • Improve and realign (towards goals) • Metrics should provide a holistic and balanced view of the business © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 24. Metrics: what is needed? • The 7 attributes of Information criteria (also known as the “IC Profile”) • Key conditions before defining a framework: • Having a pre-defined business process • Having clear goals/performance requirements • Having quantitative/qualitative measures for the business process © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 25. Metrics: Characteristics & Classification • Process Metrics • Objective/Subjective • Secure coding standards in use • Avg. time to correct critical vulnerabilities • Quantitative/Qualitative • Vulnerability metrics • Static/Dynamic • By vulnerability type • By ocurrence within a software development • Absolute/Relative life cycle phase • Management • Direct/Indirect • % of applications that are currently accepted by business partners • Trending: critical unresolved, accepted risks © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 26. Metric Specification © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 27. Metric Specification • Name of the metric • Description of what is measured • How is the metric measured • How often is the measurement taken • Range of values considered normal for the metric • Best possible value of the metric • Units of measurement © Source: Vicente Aceituno’s presentation for the FIST conferences in Madrid, 2008 © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 28. CSFs, KGIs and KPIs: what are they? • CSFs: Critical Success Factors or “vital elements” • KGIs: Key Goals Indicators or “what” has to be accomplished • KPIs: Key Performance Indicators or “how well” the process is performing © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 29. KGIs and KPIs reflect organizational goals © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 30. Example of IT metrics and KPIs • % reduction in repeat security incidents • Increased number of secure assets from risk analysis audits • % reduction of blank passwords on critical systems • % improvement on time-to-access applications • Improved bandwith use due to only-professional web surfing • % reduction in the unavailabilty of services and components (linked with corporate infrastructure management) • % efficiency improvement based on number of RFCs processed regarding vulnerabilities • % reduction in installed software not taken from DSL © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 31. Where do we show metrics?: Dahsboards and BSCs • Single point of information for infrastructure & security management • Help to make decisions and provide real-time answers to managers • Talk about the business, not about figures! • Need the involvement of the business and operations to be developed/designed in order to provide value • Web and role-based so as to get the right data (becoming the tool that consolidates siloed information) © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 32. Some dashboard examples © Business Objects. Crystal Xcelsius dashboard from www.xcelsius.com © Business Objects. Crystal Xcelsius dashboard from www.xcelsius.com © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 33. Some dashboard examples (II) © Business Objects. Crystal Xcelsius dashboard from www.xcelsius.com © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 34. Monitoring vs. Management T NG N E I M R GE TO NI A AN MO M Refine, analyze and Act on real business sort data that knowledge in a Value (and Cost) delivers security single place information according to Apply business Centralize access business need relevance to to data content information to and determine business applications priorities DATA INFORMATION ACTION KNOWLEDGE Level 2 Level 1 Level 3 Level 4 © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 35. The road to manage security information Alarm Escalation, Invoke Management Console, Response Response Model Management/Alert Management ● email ● Pager ● Cell ● ACTION Presentation Event Manage/Report Event Display, Trend Analysis, Security Reports, Performance Reports, Security System Health, Pattern Discovery KNOWLEDGE Assigning Ownership Prioritization Event Correlation Event Prioritization, Event Associations, Security Modeling Event Aggregation Log Data Reduction, Event Matching, Data Normalization and De-Duplicating Events Monitoring INFORMATION Reduction Data Filtering Event Monitoring, Third-Party Integration, Protocol DATA Data Repository Support Data Collection/Capture ● Syslog ● SNMP ● API ● © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 36. SIM and MMI Architectures query Policies Events Reporter Policy Manager Management Portal Collector ts er al Router Load Balancer SunOS Mainframe Windows X.500 Directory Router DB IDS Switch AIX Proxy Network Identity Applications /Hosts Security Systems Systems Information systems Systems © 2006 CA - All trademarks, trade names, services marks and logos referenced herein belong to their respective companies. © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 37. Using IT in the real world © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 38. Showing what really matters © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 39. Showing what really matters (II) © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 40. What can be achieved • KPIs that are a measure of how well a process is performing • The capability of predicting the probability of success or failure in the future • KPIs that are business-focused, process-oriented but IT-driven • KPIs that are expressed in precisely measurable terms • KPIs that, when acted upon, will help to improve the process • FOCUS on what is really important and has impact © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 41. The SMART side of metrics First business needs, then processes, then metrics, • then tools • Keep them simple • Use “as is/to be” & “is/is not” lists • Metrics should be S-M-A-R-T © 2008 ISACA. All rights reserved Wednesday, March 25, 2009
  • 42. THANK YOU Metrics, Measures and Myths Ramsés Gallego CISM, CGEIT, CISSP, SCPM, ITIL, Six Sigma Black Belt Certified General Manager Entel Security & Risk Management rgallego@entel.es © 2008 ISACA. All rights reserved Wednesday, March 25, 2009

Editor's Notes

  1. Let’s have a look to today’s main points in the agenda. First of all we are going to see the power of metrics and how important they are to know what is happening in a company and how the enterprise is doing regarding bottom-line impact. Metrics are the indicators that tell not only management but also people on day-to-day operations how well they are performing to already established goals and business objectives. As we will see later, there is way (and a deep need, in my opinion) to align security management with the business. We will also make a quick overview of what are CSFs, KGIs and KPIs and the intimate relationship between them. As a security practitioner and consultant, I will give you some real examples of KPIs and how they integrate in a balanced scorecard and also talk about a real implementation of a security dashboard on a customer. Finally, to wrap up, we will see the SMART side of metrics and a quick summary. Let’s go
  2. Objectives need to be defined The course in charted Risks are identified, evaluated and managed Resources and their criticality and sensitivity are determined Objectives are: Strategic alignment Risk Management Business process assurance Value delivery Resource Management Performance measurement
  3. It is said that you cannot manage what you cannot measure (and I fully agree with that vision) and my colleague Krag Brotby will later on the day do a presentation about it. It has to be pointed out that normalization of data it is very useful since you have to be able to compare between departments and divisions but also with other industry peers. Normalization places all the measures on a similar footing by equalizing them across a common organizational base Besides, metrics are rarely raw data but some derivative number (ratio, index, percentage or weighted average) Critical to successful implementation of metrics is the understanding and acceptance that they take an important commitment and use in time and resources
  4. Regarding IC, each organization needs to decide how important each attribute is for their business and this profile expresses the enterprise’s position and appetite for risk
  5. CSFs were introduced by John F. Rockart in 1979 and are defined as elements that are vital for a strategy to be successful. In another level they could also be seen as important things for the process in this way: “what you need from others” and “what you can do yourself and deliver to others” KGIs are a target to achieve, a measure of outcome We are going to focus today in KPIs since they are the day-to-day metrics, the one being monitored constantly In this context we need to remember that IT is a major enabler of the business and, therefore, KPIs are a measure of performance As you can see in the graphic on the left, KGIs are just above generic IT goals and KPIs are next to IT processes showing their area or influence. Consequently, we could define KGIs as “lag” indicators while KPIs could be “lead” indicators. By the way, both measures could also be expressed negatively showing not having reached the goal or not performing well KPIs have a cause-effect relationship with KGIs of the process In summary, KGIs are business-driven while KPIs are process-oriented
  6. I think that KGIs and KPIs do reflect organizational goals. Once a company has analyzed its mission, identified all its stakeholders and defined its goals, it needs a way to measure progress. KPIs are those measurements. Take into account that some analysts and consultants call KPIs also KSI (Key Success Indicators) but it is extremely more common the former acronym (with a P from performance) giving it a sense of direction and continuous monitoring. Top-down approach KPIs are quantifiable measurements, agreed to beforehand. However, I would like to deviate from the idea that there is a kind of negotiation with KGIs and KPIs. There should be an agreement but what really matters is the strategy and how a company is going to measure the achievement of the target. In the same way, scaling down to the IT or security department, there should be an agreement (again, not a biased negotiation) of what is needed and how security brings and adds value to the business (by preventing threats exploiting a vulnerability better than last month or year or some other measures that we are going to see in a moment).
  7. This takes us to a whole new level of data visualization and integration: dashboards and balanced scorecards. Introduced by Robert Kaplan and David Norton in the early 90s, (1992 to be precise), balanced scorecards convert strategy into action by showing in a centralized single place all the metrics that executive management needs to take decisions. In fact, not only management but also operational teams and divisional managers are empowered by balanced scorecards since different views and information is provided depending on the role and profile of the viewer. The definition of BSCs given by Mr. Kaplan and Mr. Norton is very interesting. Listen for the words: comprehensive view, performance, management tool. A BSC is a method and a management tool for ensuring enterprise’s activities in terms of its vision and strategies by giving managers a fast, comprehensive view of the performance of a business. It is here where we should introduce the 4 different perspectives of a balanced scorecard: financial, customer, internal process and learning/innovation. Scorecards - Most strategic level of the business decision while dashboard work more in the operational side giving key users metrics of their area of influence
  8. Level 0 - Non existent Level 1 - Initial Level 2 - Repeatable Level 3 - Defined Level 4 - Managed Level 5 - Optimised “Knowledge resides in the person, not in the data…it is the response and action to information that counts”
  9. We built upon other disciplines like network management, asset management (CMDB) and storage management (backup & contigency plan) so as to provide a unique repository of information and began escalating in what we called “The road to management” “You need to know what you have to be able to protect it”
  10. 3-layer architecture
  11. We focused so much in showing a KPI regarding critical operations, which nodes out of 1453 where at risk and, consequently, which operations were being threatened. Remember, at this point the definition of what is risk: the potential that a given threat will exploit a vulnerability with an impact in an asset or group of assets
  12. (meaning alignment with the business) (since KPIs are “lead” indicators) FOCUS
  13. SIMPLE MEASURABLE ACHIEVEABLE REALISTIC TIME-DRIVEN