Downloaded 31 times
Uploaded byAleksey Lukatskiy
ICS Cyber Security Effectiveness Measurement
The document discusses the importance of measuring cybersecurity effectiveness in industrial control systems (ICS), emphasizing the distinction between operational, tactical, and strategic metrics. It highlights various metrics examples, best practices, and categorical impacts from cybersecurity incidents while also outlining how businesses can assess their vulnerability and improve their defenses. Additional focus is placed on aligning cybersecurity efforts with business goals to enhance overall organizational security and resilience.
More Related Content
![Overview of the Cyber Kill Chain [TM]](https://cdn.slidesharecdn.com/ss_thumbnails/cybersecuritykillchain8-161209191549-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to ICS Cyber Security Effectiveness Measurement
![[CONFidence 2016] Gaweł Mikołajczyk - Making sense out of the Security Operat...](https://cdn.slidesharecdn.com/ss_thumbnails/gawemikoajczyk-makingsenseoutofthesecurityoperations-160601113717-thumbnail.jpg?width=640&height=640&fit=bounds)
ICS Cyber Security Effectiveness Measurement
- 1. Alexey Lukatsky Security businessdevelopment manager ICS Cyber Security Effectiveness Measurement
- 2. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Not Petya / Nyetya Tools Tactics • Supply chain and victim to victim pivoting • Rapid Infection Spread • Destroyed Countless Systems / Networks Processes • Designed to inflict damage as quickly and effectively as possible. • Appears to be Ransomware, but is purely destructive • Wormable Ransomware • Designed to Spread Internally Not Externally • Leveraged Eternal Blue / Eternal Romance and Admin Tools (WMI/PSExec) • Advanced Actor associated with a Nation State • Destructive Attack Masquerading as Ransomware • Most Expensive Incident in History Description
- 3. ICS Kill Chain 1 2 3 4 5 6 7 8 Conficker APT1 Иранvs США BE3 HAVEX Stuxnet Ukraine 2016 WannaCry Neytya
- 4. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Why we need to measure our effectiveness? • Good security not visible • We want to show that we work well • Top management often wants to compare itself with others • We want to see the dynamics
- 5. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Rare Remote Possible Likely Very likely Catastrophic 6 7 8 9 10 Significant 5 6 7 8 9 Moderate 4 5 6 7 8 Minor 3 4 5 6 7 Insignificant 2 3 4 5 6 Accept (score = 2,3) Monitor (score = 4,5) Manage (score = 6) Avoid / Resolve (score = 7) Urgently avoid/ Resolve (score = 8, 9, 10) “Best practices” for security measurement • Not specifically, not quantitatively, conditionally… Impact Probability
- 6. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public
- 7. Cybersecurity is stateof protection of the interests of enterprise stakeholders in the information area, determined by the totality of balanced interests of the individual, society, state, and business Or process? Not important!
- 8. Efficiency/effectiveness is the quantifiable contributionto the achievement of ultimate goals
- 9. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public What goals can we have? • Fulfillment of NERC CIP or ISA/IEC 62443 requirements • Categorization of all CI objects • Certification of key processes for ISO/IEC 27019 • Reduce the number of ICS cybersecurity incidents to 3 per month • Implementation of secure remote access to ICS for contractors • Reduce downtime from ICS cybersecurity incidents to 2 hour on average • Cost reduction for ICS cybersecurity for 15%
- 10. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public • Operational (наиболее привычные) • Realtime, day-to-day • Logs, rules, signatures, etc. • How effective is your security measures? • Tactical • Change control • Scorecards and audits • How effective is your security program? • Strategic • Corporate risk and business alignment • How are we secure? Strategic Tactical Operational Measurements are different
- 11. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Tactical metrics examples • Incidents requiring manual cleanups • Mean-Time-to-Fix • Also TTR (Time-to-Recovery) or TTC (Time-to-Contain) • Mean-Time-to-Detect • Mean-Time-to-Patch • Involvement of staff in cybersecurity activities • Mean cost to mitigate vulnerabilities
- 12. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Tactical metrics examples • % of ICS without known severe vulnerabilities with CVSS >7.0 • % of changes with security review • % of changes with security exceptions • ICS cybersecurity budget allocation (% of total, IT, cybersecurity, ICS) • Compliance rate • Cost of incidents
- 13. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Tactical metrics examples • Time between creating and closing a ticket for an incident • Ratio of open and "closed" incident reports • Ratio of incidents and tickets • Number of repeat incidents • Ratio of communication methods (e-mail / calls / portal) • Number of false positives (non-existent incidents)
- 14. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public SMART principle for metrics selection • SMART – Specific, Measurable, Achievable, Relevant, Timely • As concretely as possible, without double interpretations, for the right target audience • The result should be measurable, not ephemeral • Why choose a goal that is unattainable? • Relevance to goals • Timeliness and relevance
- 15. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public SMART usage example for ICS Cybersecuirty Characteristic Example of bad metric Example of good metric Specific The number of failed login attempts to the HMI The number of failed login attempts to the HMI for one week for one employee Measurable Income from the implementation of an ICS cybersecurity The employees loyalty level about ICS Cybersecurity Achivebale The absence of cyber security incidents in ICS for the current quarter The number of ICS cybersecurity incidents in the current quarter <5 Relevant The number of opened projects for ICS cybersecurity The number of completed on time projects for ICS cybersecurity Timely The number of patched ICS nodes last year The number of unpatched ICS nodes current year
- 16. How to movefrom hundreds of operational metrics to one or two strategic?
- 17. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public From individual metrics to measurement program • EPRI (Electric Power Research Institute) Research Program • Creating Security Metrics for the Electric Sector (Parts I, II, III, IV) • Applicable to a wide range of industrial enterprises outside the electric power industry 3 strategic metrics 10 tactical metrics 45 operational metrics
- 18. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public From individual metrics to measurement program Strategic Metric Name Tactical Metric Name Protection Score Network Perimeter Protection Score Endpoint Protection Score Physical Access Control Score Human Security Score Core Network Vulnerability Control Score Core Network Access Control Score Data Protection Score Security Management Score - Protection Detection Score Threat Awareness Score Threat Detection Score Security Management Score - Detection Response Score Incident Response Score Security Management Score - Response
- 19. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public From individual metrics to measurement program Tactical Metric Name Operational Metric Name Network Perimeter Protection Score Mean Access Point Protection Score Mean Wireless Point Protection Score Mean Internet Traffic Protection Score Mean Count-M Malicious Email Mean Count-M Malicious URL Mean Count-M Network Penetration Security Management Score - Protection Security Budget Ratio Security Personnel Ratio Cybersecurity Risk Tolerance Score
- 20. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public From individual metrics to measurement program Operational Metric Data input to the Formula Mean Access Point Protection Score Number of inbound connections per day Number of dropped inbound connections per day Number of all alerts per day Number of security alerts per day Number of probes per day Number of confirmed DOS attempts per month Чnumber of confirmed intrusion attempts per month Number of confirmed incidents that required human intervention per month
- 21. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Automation tool: EPRI MetCalc
- 22. What does thebusiness think of all these metrics?
- 23. Business thinks aboutcybersecurity, but in its own way Reservoir Pump Water intake Water treatment plants Underground tank Pump Distribution Cleaning with reagents, ozone and coal Sump Flats / Houses Water meter Smooth operation Correctand uninterrupted bills Smooth operation Continuous diagnosis Telemetry control Continuous monitoring Proper dosing FZ-152 Order №31 CIP Law Water supply process
- 24. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public The difference in the perception of top management and cybersecurity / IT / ICS Cybersecurity / IT / ICS • Deep dive to details • Unwillingness to share collected data • Data for data, not for decisions • What? Where? When? Top management • Bird's-eye view • Data for decision making • What will happen? What to do?
- 25. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Time Productivity 0 20 40 60 80 100 А В С ВТ Т1 Т2 Т3 D = System failure / disaster R = The possibility of attenuating or mitigating the effect before or during a negative event A = The ability to absorb and degrade В = Lower limit; threshold value ВТ = Lower limit duration С = Ability to return to baseline D → R How does a business see security incidents? Reduce А? Reduce Вт? Reduce С? Reduce Т1, Т2 and Т3?
- 26. Let's try toreformulate our goals Profit increase Geo expansion Sales increase Production optimization Reduction in logistics costs Loss reduction X hours of downtime due to ransomware Y hours of process downtime due to DoS/DDoS-attack Z hours of employee downtime due to spam N rubles fine from supervisory authorities Business Cybersecurity
- 27. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public From the “for myself” measurement to the measurement for business 75% 55% Q2 Q1 The number if incidents by sources The number of ICS incidents Downtime Incidents dynamics Contracts loss $35M127
- 28. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Cybersecurity incidents loss types Productivity •Downtime •Deterioration of the psychological climate Response •Incident forensics •PR-activity •Support Service Replacement •Equipment replacement •Re-entry of information Fines •Legal costs, pre-settlement •Suspension of deals Competitors •Know-how, commercial secrets •Customer churn, overtaking by competitors Reputation •Goodwill •Decrease in capitalization, stock price Other •Rate downgrade •Decrease in profitability
- 29. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Impact categories Insignificant Minor Moderate Significant Catastrophic Finance impact of more than $Y $1М $5М $10М $50М $100М Let's be more specific and measure the money • The cost of direct losses from disruption of business operations • Business Transaction Recovery Cost • Decrease in stock prices (dumb indicator, but sometimes also measurable) • Fines • Lost profit (if you can count it) • Decrease in customer loyalty • Replacing equipment or re-entering information • Interaction with affected customers, etc.
- 30. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Questions for defining strategic business metrics of cybersecurity • What will stop or slow down operations in your organization? • What will lead to a decrease in profits / revenue / margin / market share of your company? • What will lead to a decrease in the quality of the product / service? • What will lead to a negative impact on the goal of the company / business unit / business project / executive sponsor?
- 31. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Impact categories Insignificant Minor Moderate Significant Catastrophic Outage of more than X customers 10 customers 100 customers 500 customers 1000 customers 5000 customers Business operations disruption of >= Z min / hours / days 1 hour 4 hours 8 hours 2 days 5 days Serious injury to >= A people 0 people 0 people 1 person 10 people 50 people Breach of data for >= B customers 100 customers 1000 customers 5000 customers 10000 customers 100000 customers Loss of >= C customers 5 customers 10 customers 25 customers 50 customers 100 customers Loss of market share for D% 0% 0% 1% 3% 7% Productivity loss for E% 0% 1% 3% 5% 10% If you can’t count in money?
- 32. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public The duration of an cybersecurity incident in terms of cybersecurity and business § The influence level and price components of an incident changes over time This illustration can be used to estimate recovery time after an attack RPO – Recovery Point Objectives, RTO – Recovery Time Objectives, MAD – Maximum Allowable Downtime
- 33. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Impact categories Insignificant Minor Moderate Significant Catastrophic Reduction of power generation by F megawatts Power reduction is acceptable Power reduction is acceptable 100 MW 1000 MW 10000 MW Impact categories Insignificant Minor Moderate Significant Catastrophic Publications in mass media Absent In local consumer print media On local TV or in local industry publications On national TV or in national consumer print media Highlighted broadcasts or reporting on national TV or in national industry print media Industry specific metrics
- 34. How to measure cybersecurityfor a business, but not with money?
- 35. Can compare yourselfwith competitors? 0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 План & бюджет Организация Защитные меры Архитектура Процессы и операции Осведомленность Реагирование Управление уязвимостями Оценка рисков Корпоративное управление В среднем по отрасли У нас Tricks: instead of comparing with competitors (if there is no data), you can compare yourself in different states (there was - now - in a year - ideal)
- 36. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public 5 important metrics • % of cybwersecurity activities unlinked to business goals • Number of projects / activities linked to business goals • % of projects / assets / services that are important for business that do not meet cybersecurity requirements • For example, uncontrolled remote access by contractors • % of projects / assets / services that are important for business and whose security measures are inadequate or ineffective • Or for whom during the incident the response plan did not work • The likelihood of providing services during an cybersecurity incident You can still play with the risks ...
- 37. © 2018 Ciscoand/or its affiliates. All rights reserved. Cisco Public Common errors in effectiveness measuring • Choosing hundreds of metrics instead of focusing on strategic • Measuring what is easier to measure instead of focusing on measurement goals • Lack of business focus • Focus on operational result-oriented metrics instead of evaluating process performance • Lack of context • Cybersecurity price reduction with incidents growth
- 38. Key Success Factors •You must understand what you are doing in the field of information security • You must understand your business • You must understand your target audience • You must be able to combine these three elements together • You need to know where the data is • You must be able to code/program
- 39.
- 40.