Measuring the effectiveness of an information security program is important for governance, continuous improvement, and providing assurance. Key things to measure include compliance with frameworks, control effectiveness, and progress towards goals. Metrics should be tailored to different audiences like executives, managers, and security staff. Example metrics include vulnerability remediation timelines, audit findings closure rates, and security event trends over time. Visual dashboards with indicators like colors and arrows help concisely communicate security program status and areas needing attention to stakeholders.

1. Demonstrating Information
Security
Program Effectiveness
Doug Copley
CISO – Beaumont Health
Chairman Emeritus – Michigan Healthcare
Cybersecurity Council
16-SEP-2015

2. Why Measure your InfoSec Program
?

3. Why Measure your InfoSec Program
» Integral to program governance
» Necessary to measure process controls
» Can determine control effectiveness
» Can show resource gaps or shortages
» Core to measuring service provider SLAs
» Necessary for continuous improvement
» Provides assurance to executives & the Board
“If you can't measure it, you can't manage it”
- Peter Drucker

4. What Should I Measure?
» Depends on Audience
 C-level executives or the Board
 Management stakeholders
 Information Security leaders
 Information security staff
» Metrics must be meaningful to the audience
 They will be different by audience
 There is no one metric that will fit all of them

5. What Should I Measure?
» Begin with a baseline
» Baselines are important to measure progress
» Leverage your security framework structure:
 ISO 27001
 NIST
 HiTrust
 SANS 20 Critical Controls
» Customize the structure to fit your program

6. Example Program Scorecard

7. Example Program Scorecard
SANS 20 Critical Security Controls

8. Example Program Scorecard
InfoSec
Management
Program (IS)
Access Control
(AC)
Human
Resources
Security (HR)
Risk
Management
(RM)
Security Policy
(SP)
Organization of
Information
Security (OI)
Compliance (CO)
Asset
Management
(AM)
Physical Security
(PS)
Communications
Security (CS)
Systems
Acquisition,
Development, and
Maintenance (SD)
Incident
Management
(IM)
Business
Continuity (BC)
Cryptography
(CR)
Operations
Management
(OM)
Supplier
Relationships
(SR)
4 22 2 4
3 3 2
0 5
11
0 1
1 2 7 0
Modified ISO 27001 Scorecard

9. HIPAA Compliance

10. Example Metrics
Business Impact Analysis (BIA) – 2016 required
Incident Management Plan Updates/Testing – 2015 required
Call Tree Testing – 2015 required
Power Outage Plan Test – Troy Data Center – 2015 required
Pending Tasks
SunGard Assurance User Training – 2016 required
14
32
18
37
8 7
9
7
12
30
17
37
0
5
10
15
20
25
30
35
40
Tier 0 Tier 1 Tier 2 Tier 3
Total Critical Apps App Plans Updated 2014/2015 Apps Not Tested in 2014/2015
Critical Application Status
Business Continuity

11. Vulnerability Management

12. Vulnerabilities and Malware
Vulnerabilities Malware

13. Periodic Security Risk Assessment
» Can provide a risk baseline
» Can provide an estimated compliance baseline
» Provides process to measure progress
» Must consider all “reasonably foreseeable risks”
» Should have close alignment to regulatory
expectations and guidance
» Make sure scope is complete so you don’t end
up doing another one to catch areas
» Will be primary input into security roadmap

14. Executive Metrics

15. Governance Committee Metrics

16. IT Management
Metric Count Frequency
Category2 Category3
Short or
Long
Term Description Monthly Quarterly Semi-Annual Weekly Grand Total
Access Mgmt CA Coverage L % of applications FIM integrates with 1 1
Assessments 3rd Party S # completed 3rd party assessments 1 1
L % partner agreements with approved security reviews 1 1
Project L % projects approved by security prior to implementation 1 1
Asset Mgmt Backup L % backup tapes encrypted 1 1
Audit/Exam Actions Required S # open action plans 1 1
Audits S # audits completed 1 1
Benchmark Desktop/Laptop L % of end user PCs with non-standard software installed 1 1
% of end user PCs with standard build 1 1
Configuration Coverage L % of devices being scanned for configuration issues 1 1
Fix Time L Average amount of time to apply fix by criticality 1 1
Overdue L Overdue Critical, High, Medium, Low config defects 1 1
Volume L # of open configurationitems needing to be resolved 1 1
Finance Budget L Budget allocated to IS sustain and projects 1 1
Internet Use Exceptions S # approved exceptions 1 1
# exceptions by LoB 1 1
User Activity S % locations using global filteringrules 1 1
Intrusions SEIM S Suspect attacks 1 1
Investigations incidents S # lost/stolen PCs 1 1
L # incidents by incident type 1 1
# operations-reportedincidents 1 1
# user-reported incidents 1 1
Logging SEIM L % customer-facingand internet-facingsystems with monitored event and activitylogs (SEIM device) 1 1
% systems with monitoredevent and activity logs (SEIM device) 1 1
Malware All platforms L malware detected on servers, desktops and laptops 1 1
Email L Detection failure rate 1 1
Patches Fix Time L Average amount of time to apply fix by criticality 1 1
Overdue S Overdue Critical, High, Medium, Low patches 1 1
L Overdue Critical, High, Medium, Low patches 1 1
Perimeter Firewall S Volume of changes 1 1
L Inbound connections to internet-facingservers 1 1
Number of locations connected to the core network without intermediatefirewalls 1 1
Wireless L Rogue Access Points detected 1 1
Personnel Education/Training S # communications/period 1 1
# training sessions offered/period 1 1
% employees affirmedto AUP 1 1
% security staff with professional security certifications 1 1
L % employees taken required IS & Privacy training 1 1
% non-employees affirmed to AUP 1 1
Utilization S Overtime Reports 1 1
IS Time by Category 1 1
Policy Conformance L % criticalassets on compliancesystems 1 1
Non-conformance by Hosting Provider 1 1
Total # of non-complianceissues by priority 1 1
Exceptions S Exceptions by platform 1 1
Risk Acceptance S Risk Acceptance by policy 1 1
Risk Acceptance by platform 1 1
Transmission 3rd Party L % externaldata sharing protected with confidentialityand integritycontrols 1 1
Email L % suppliers and vendors with mandatory TLS enabled 1 1
Vulnerabilities Coverage S % of devices being scanned for vulnerabilities 1 1
Fix Time L Average amount of time to apply fix by criticality 1 1
Process Maturity S Process Maturityvalues by self-assessment 1 1
Information Data Loss Prevention S Data in transmission events by severity 1 1
Data at rest events by severity 1 1
Data at endpoint events by severity 1 1
Grand Total 23 23 7 2 55

17. Information Security Team
Board (All)
GRMC (All)
IT Mgmt (All)
Unique Metrics
Row Labels Total
Governance 37
Assessments 3
Asset Mgmt 2
Audit/Exam 3
Customer Interface 1
Finance 1
Personnel 14
Policy 12
Process 1
Incident Response 8
Forensics 2
Investigations 6
Operations 28
Configuration 4
Internet Use 7
Intrusions 2
Malware 6
Perimeter 5
Vulnerabilities 4
Protection 25
Access Mgmt 7
Assessments 1
Benchmark 3
Information 3
Logging 2
Malware 3
Patches 4
Transmission 2
Grand Total 98

19. Ways to Organize Metrics
» Slide libraries (demo)

20. Creating Meaningful Diagrams
» A good security metric can provide the
following information:
 Current value – normal or abnormal
 Current value – satisfactory or unsatisfactory
 Trend – Increase or decrease
 Trend – Improving or degrading
» Use colors and arrows to represent them in a
compact and concise way
» Ideally, it will clearly show required actions

21. Making Them Visual
» Make the slides visually appealing and
informative to the target audience

22. Show Metrics in a Time Series
48,953
106,971
56,602 57,801
44,808
62,262
52,210
39,460
52,210
36,297
45,256
0
21,111
0
20,000
40,000
60,000
80,000
100,000
120,000
AUG 14 SEP 14 OCT 14 NOV 14 DEC 14 Jan-15 Feb-15 Mar-15 Apr-15 15-May 15-Jun 15-Jul 15-Aug
Attempts to Access Malware

23. Improving or Worsening?
547
117
183
45
418
48
274
346
196
184
230
64
172
116 89
219
58
68
-
100
200
300
400
500
600
Mar-14 Apr-14 May-14 Jun-14 Jul-14 Aug-14
Data Loss Prevention Security Events (n=193)
Only IronKey SSN US Credit Cards Linear (Only IronKey) Linear (SSN) Linear (US Credit Cards)

24. Does Downward Slope Mean Good?
3
0 1
67
0
7
0 1 1 0 0 0
10
1 1 0
9
2 2 0 00
4
1 1
7
19
3
0 0 1 1 2 0 0 1 2 0 1 0
20
6665
61 61
127
120
108
105 106 107 106 105 103
113 114 114 112
121 122 124
104
38
0
20
40
60
80
100
120
140
2012-08
2012-09
2012-10
2012-11
2013-01
2013-02
2013-03
2013-05
2013-06
2013-07
2013-08
2013-09
2013-10
2013-11
2014-01
2014-02
2014-03
2014-05
2014-06
2014-07
2014-08
IT Audit and Compliance Items - 24 Month Trend
New
Closed
Open

25. Pulling It All Together
» Do NOT make metrics and dashboards
slideware that is shared via email
 Make sure you can explain them in person
» Make sure the slides not only show the data,
but explain the significance
 Help the reader understand what to look for
» Produce/present them regularly
 Need to be consistent to develop credibility
» Answer the question “So where do we stand
with our information security program now?”

26. Executive Dashboard
» Intended to convey a high-level status of the
program to C-level executives and the Board
» Security Dashboard should convey:
 Status of regulatory compliance
 Capability, Maturity and Implementation level of
program elements
 Key areas of information risk to the organization
 Current initiatives and future state posture
 External ties and intelligence information
» Must answer the question “Is our Information
Security program effective?”

27. Sample Dashboard #1
27

28. Sample Dashboard #2
28

29. Sample Dashboard #3

30. Sample Dashboard #4

31. Questions?

32. Thank You!