You're the newly-minted CISO in your organization, charged with the (un)enviable task of improving security. Unfortunately, your superiors and peers aren't quite sure what good security looks like, what they expect to see from you, or how you should go about doing it. All the execs know for sure is that if a security breach happens, it’s definitely your fault...and if a breach never comes, they question the need for security’s budget. It falls to you to chart your own path, and help define what good security looks like within your organization. In this session we will provide real-world examples of how the three speakers have faced this challenge in multiple organizations, what metrics were chosen to show progress, and how the speakers have gone about gathering them. You will leave this session not with abstract ivory-tower ideas on measurement, but with actionable tactics you can put in place within your own program today. This session will address: • How to show security progress • Presenting security to senior leadership • Real-world security metrics • Identifying and using easily collected data • Aligning with existing organizational metrics

1. MOVING MOUNTAINS THROUGH MEASUREMENT
Chris Clymer
Director of Security
MRK
Jack Nichelson
Director of Infrastructure & Security
Chart Industries
Jason Middaugh
Director of Infrastructure& Security
Cliffs Natural Resources

2. INTRODUCTION
 Why are we here?
 What are our goals?
 What will you gain from this presentation?

3. WHAT WILL YOU GET FROM THIS PRESENTATION?
 A repeatable process for measuring security performance
 A playbook you can start executing on tomorrow
 Methods for explaining security to your CEO
 Tools for justifying security budget

4. WHAT WONT YOU GET FROM THIS PRESENTATION?
 Our favorite metrics
 Details on how to build the best spreadsheet
 Instructions on using fancy reporting tools

5. CHRIS CLYMER
 Director of Security Services for MRK
 CISO for companies ranging from SMB’s to
multi-billion dollar corporations
 Former board member for NEOISF & co-
host of the Security Justice podcast
 Aspiring Ironman, amateur saberist

6. JASON MIDDAUGH
 Director of Infrastructure & Security Services at Cliffs
Natural Resources
 Holds a bachelor’s degree in information
systems/operations management from the University
of Toledo, an executive master’s degree in business
administration from The Ohio State University, and
holds several IT based certifications (CISSP, MCSE,
VCP, CCNA, CCDA).
 Chair member of the North East Ohio Cyber
Consortium. Selected and currently attending the FBI
Citizens Academy.
 Skiing enthusiast

7. Jack Nichelson “Solving Problems, is my Passion”
 Director of Infrastructure & Security for Chart Industries.
 Executive MBA from Baldwin-Wallace University
 Recognized as one of the “People Who Made a Difference
in Security” by the SANS Institute and Received the
CSO50 award for connecting security initiatives to
business value.
 Adviser for Baldwin Wallace’s, State winner Collegiate
Cyber Defense Competition (CCDC) team.
I defend my companies competitive advantage by helping solve business problems through
technology to work faster and safer.
JACK NICHELSON

8. WHY METRICS?

9. WHY METRICS?
 Can only improve what you can measure
 Without measurement, security is purely artistry and witchcraft
 Because security wants a seat at the big table
 Business units that report to the board will have some way to demonstrate their performance
 Because you need to know what’s working, and what needs your
attention
 Because you want to buy more security toys next year
 Because you want to add more staff

10. HOW I’VE USED THEM
 Drive team performance
 Explain security to the board
 Justify past expense
 Justify FUTURE expense
 Keep myself honest

11. HOW I’VE USED THEM
4,600 8,190 6,200 6,900 9,100 9,100 4,600 4,800 7,600 8,425 7,532
16,968 15,786 15,865
26,661 30,449
284
13,893
2,018 2,009 2,675 3,610 4,009
62 103
638
1,727
3,030
35,864 62,404
105,617
112,804
125,653 132,645
11,009 15,900 17,006
16,101
20,782
14,680
12,910
63,501
55,696
10,850
84,630
90,752
14
64 34
12
17
25
46
32
11
11
8
5
32,591
40,002 39,208
50,312
62,075
18,019
71,813
132,555
168,133
133,940
222,326
234,943
0
50000
100000
150000
200000
250000
Dec-14 Jan-15 Feb-15 Mar-15 Apr-15 May-15 June-15 July-15 Aug-15 Sept-15 Oct-15 Nov-15
Total # of New Security
Incidents
Total # of Malicious
Websites Auto-Blocked
Attacks Auto-
Prevented by IPS
(Critical/High)
Total # of Malware
Auto-Handled
Total # of Phishing
Auto-Handled
• IPS Tuned
• AV
Signature
Issue
• New Web Filter

12. HOW I’VE USED THEM
4,600 8,190 6,200 6,900 9,100 9,100 4,600 4,800 7,600 8,425 7,532
16,968 15,786 15,865
26,661 30,449
284
13,893
2,018 2,009 2,675 3,610 4,009
62 103
638
1,727
3,030
35,864 62,404
105,617
112,804
125,653 132,645
11,009 15,900 17,006
16,101
20,782
14,680
12,910
63,501
55,696
10,850
84,630
90,752
14
64 34
12
17
25
46
32
11
11
8
5
32,591
40,002 39,208
50,312
62,075
18,019
71,813
132,555
168,133
133,940
222,326
234,943
0
50000
100000
150000
200000
250000
Dec-14 Jan-15 Feb-15 Mar-15 Apr-15 May-15 June-15 July-15 Aug-15 Sept-15 Oct-15 Nov-15
Total # of New Security
Incidents
Total # of Malicious
Websites Auto-Blocked
Attacks Auto-
Prevented by IPS
(Critical/High)
Total # of Malware
Auto-Handled
Total # of Phishing
Auto-Handled
• IPS Tuned
• AV
Signature
Issue
• New Web Filter

13. WHAT MAKES A GOOD METRIC?
 Easily measured
 Easily understood
 Enables decision-making
 Meaningful
 Consistent
 Quantitative
 Aligns with your
organization’s Risk Profile

14. MY PROCESS
1. Define the problem
2. Identify your resources
3. Build a rough draft
4. Review with stakeholders
5. Rinse & repeat

15. MY PROCESS
1. Define the problem
a. Where is the pain?
b. Find your stakeholders
c. What is working/what isn’t?
2. Identify your resources
a. What is readily available?
b. Tools, process, people
c. Don’t dwell on nice-to-haves
3. Build a rough draft
a. Won’t be perfect the first time
b. Make sure its easy to repeat
4. Review with stakeholders
a. Above and below
b. Have them contribute
c. Get them to buy-in
5. Rinse & repeat
a. Run the process
b. See what works and what doesn’t
c. Adjust where needed

16. JASON'S STORY

17. FIRST STEPS – GETTING STARTED
 Know your audience and speak their language
 Leverage existing tools that you already have
 Anticipate what others will ask or what conclusions they
will draw from the metrics you are presenting

18. PAST PROJECTS AND SPEND
 IT Executives and Board members want to see that they getting results from
previous initiatives
 Use these slides to create a “Wow” factor

19. CURRENTLY MEASURED METRICS
 Your goal should be to eventually get to at least 13 months of revolving data
 Play the Game of Thrones, or in this case the Game of Met-tricks

20. LEVERAGE METRICS FOR YOUR FUTURE NEEDS
 Use your metrics to help
justify additional capital
 Help predict when you are
going to run out or need
more
 Show capability gaps that
you want to fill
 Justify additional
resources

21. HOW AND WHERE DO I GET STARTED?
 Kick things off in your next team meeting
 Don’t worry about the past
 Don’t get hung up on 100% accuracy
You don’t have to hit the bulls-eye on every metric, shoot for the inner-rings

22. SUMMARY
 Know your audience - Speak their language
 Think Past – Leverage metrics to justify and
put historic spend at ease
 Think Present – Collect and analyze your data;
do not be afraid to burn it down and start again
 Think Future – Continually set the stage and
constantly communicate when and where you
are going to need additional resources with
metrics
 Get Started! – You do not have to hit the
bullseye

23. JACK'S JOURNEY

24. BE PROACTIVE
Change starts from within, so you have to make the decision to focus
on the things you can influence rather than reacting to the things
outside of your control.
Manage Yourself:
 Where and how are you spending your time & energy throughout the day?
 Make a list of the things that concern you and things you can Influence.
Ask yourself these 3 questions every day:
 Did I do my best to spend my time on things I can influence?
 Did I do my best to set and communicate clear goals?
 Did I do my best to make progress toward goal achievement?
“The 1st metric you need to track is yourself”

25. BEGIN WITH THE END IN MIND
If your ladder is not leaning against the right wall, every step you
take gets you to the wrong place faster.
First, do you know what “good” looks like?
 Break down the area you have influence over into functional parts
that you and the stockholders can score and rank.
 Now that you have an agreed upon heatmap of your current state,
set short term and long term goals.
“Try Not to Become a Success. Rather Become a Person of Value.”

26. PROBLEM STATEMENT
The Problem Statement significantly clarifies the current situation by
specifically identifying the problem and its severity, likelihood, and
impact. It also serves as a great communication tool, helping to get
buy-in and support from others.
Build & Execute plans to drive for results & share successes
 Invest more time in project planning and due diligence; time spent defining
the problem is NEVER time wasted.
 Write a Project Charter, clearly state the scope, objectives, participants, and
success measurements.
 Create a Work Breakdown Structure to graphically represent the project
scope, broken down in successive chunks with defined deliverables.
“People with a plan succeed because they know where they’re going”

27. PUT FIRST THINGS FIRST
Focus on the important, not just the urgent. The urgent are not that
important, and the important are never urgent.
Tips for taking back control of your time:
 Stop saying Yes, When you want to say No.
 Scheduled your own time with purpose & defend it!
 Don’t be afraid to close your email and turn off your phone
“Effectiveness requires the integrity to act on your priorities”

28. CHART PERFORMANCE & ADJUST
Gemba (現場) is a Japanese term referring to the place where value is
created. The idea of Gemba is that the problems are visible, and the
best improvement ideas will come from going to the Gemba.
“Good security is not something you have, it’s something you do” - Wendy Nather

29. SUMMARY – KEY TAKEAWAYS
 Be Proactive – Focus on what you can influence
 Begin with the end in mind – Define practical outcomes
 Create a Problem Statement – A goal without a plan is just a wish
 Put first thing first – Plan weekly, act daily
 Chart Performance & Adjust – Shine a light on the problem
“Think about how you can simplify security – make it easy – and focus on the basics.” - Dave Kennedy

30. WRAPPING UP

31. OUR 3 APPROACHES
Chris Jason Jack
1.Define the problem
2.Identify your resources
3.Build a rough draft
4.Review with
stakeholders
5.Rinse & repeat
1. Know your audience
2. Think Past
3. Think Present
4. Think Future
5. Get Started!
1.Be Proactive
2.Begin with the end in
mind
3.Create a Problem
Statement
4.Put first thing first
5.Chart Performance &
Adjust

32. THANK YOU
Chris Clymer
Chris@ChrisClymer.com
@ChrisClymer
Jack Nichelson
Jack@Nichelson.net
@Jack0lope
Jason Middaugh
Jason.Middaugh@CliffsNR.com
@Jason_Middaugh

33. APPENDIX

34. REFERENCES
 Security Metrics – Andrew Jacquith
 Security Data Visualization – Greg Conti
 Pragmatic Security Metrics – W. Krag Brotby & Gary Hinson
 Security Metrics Mailing list - http://www.securitymetrics.org/mailing-list.html
 NACD Cyber Risk Oversight Handbook 2016 - NACD

35. HOW TO AUTO-LINK EXCEL GRAPH (OR CELLS) TO POWERPOINT DECK
 Open Excel Click on Graph (or Cells) and hit Copy
 In PowerPoint select Paste  Paste Special …
 Change the radio button to “Paste link” and highlight “Microsoft Excel Chart Object” and
press OK
 The Chart (or Cell) with appear in the PowerPoint Deck and with auto-update from the
source Excel file each time the PowerPoint Deck is opened
Notes:
 I highly recommend you save both the Excel and PowerPoint files in shared workspace
like SharePoint
 Do not rename the Excel file, it will break the links
 Do not move the location of the Excel file, it will break the links
 Do not rename the tabs in the Excel file, it will break the links
 If you delete and recreate the graph in Excel, you will need to re-link it to the PowerPoint
presentation (the graph will have a new name)

36. HOW TO BUILD A SQCD BOARD
 Key Performance Indicators – Good data can tell a story
 Predictive Analysis – Your board should help prevent future issues
 Keep the data fresh and useful, address items as quick as possible
using LEAN tools and once addressed remove them from the board.

37. GEMBA BOARD: SECURITY
“We measure things that matter”
Example Metrics:
 # of systems not monitored & tracked in inventory by
Location or LoB
 # Top Vulnerabilities by Location or LoB
 # of Legacy Systems by Location or LoB
 # of Users with Local Admin & Accounts with Domain Admin
 # of Total Security Incidences by Location or LoB
 # of Past Due Security Awareness Training by Location or
LoB
Security - The current security posture at a glance

38. GEMBA BOARD: QUALITY
Example Metrics:
 # of Servers & Workstation missing OS & App patches
(30 day SLA)
 # of infections/Re-Images tickets (3 day SLA)
 # of Security Event tickets (5 day SLA)
 # of Security Request tickets (15 days SAL)
 Cause Mapping Analysis to find root cause of problems
Quality – Results for SLA goals of
events & requests

39. GEMBA BOARD: DELIVERY
Delivery – Active Projects & Audits at a glance
Example Metrics:
 Active Projects Status
 Active Audit Status
 Remediation Progress by Location or LoB
 On-Site Awareness Training by Location

40. GEMBA BOARD: COST
Cost – P&L at a glance
Example Metrics:
 Operating budget spending plan (OPEX & CAPEX)
 ROIC Qualitatively Rating of Perceived Value
 Support Agreements Costs & Renew dates
 Consultant Support Agreements Costs & Renew dates
 Running total of cost savings

41. GEMBA BOARD: PEOPLE
People – Skills matrix at a glance
Example Metrics:
 Skills Matrix of everyone in Security
 Training and development plans
 On-Call & Vacation Schedules
 Awards

42. VISUALIZATION TECHNIQUES: THE HEATMAP
Impact
Low No threat to core business function impact
Medium
Threat to core business function impact, but has
not occurred yet. i.e. ERP system is down but
have not yet missed orders
High
Immediate impact to core business functions. i.e.
products cannot be shipped, or core IP is lost.
Likelihood
Low Happens once every 10 years, or less
Medium Happens once every 1 to 10 years
High Happens once or more a year
• Develop “Likelihood” to fit your org
• Develop “Impact” to fit your org”
• Score potential risks “high”,
“medium”, or “low” for each
• Map results to the heatmap

43. VISUALIZATION TECHNIQUES: RISK REGISTER

44. VISUALIZATION: MULTI-LAYERED DEFENSE STRATEGY
95% of systems are patched
within 30 days.
Once discovered - improve tools.
Block about 27,500 intrusions & 880 virus / month
Detect & fix 20 viruses & 3,500 exploits / month
Block 1.6 million emails & Stop 22,000 virus / month
Successful
Unknown Attacks
IPS
Spam Filter
Admin Rights Removal
Anti Virus & MalwareBytes
Employee User Security Training
Hardware refresh & Laptop Encryption
Password management
Patching strategy
Outside independent measurement of security outcomes
Block 1.6m /month bad IP’s
and 600,000 / month bad URLs
C
o
n
t
a
i
n
P
r
e
v
e
n
t
Independent penetration testingAttackers
BitSight
Score
Firewall / Open DNS IP Filter / Web Filter
2600 Users no longer have Admin Rights
97% of Laptops are now encrypted

45. VISUALIZATION TECHNIQUES: CLUSTERED STACKED GRAPH
3
4
3 3
4
5
4 4 4 4
6
3
3
2
2
4
4
3
4
2
6
8
8
8
8
8
7
6
6
2
2 2
2
2
2
2
2
2
2
2
17 17
16
15
16
19
17
15
16
10
0
5
10
15
20
# of Open Low Vulns
# of Open Moderate Vulns
# of Open Important Vulns
# of Open Critical Vulns
 Good for showing
portions of a whole
over time
 Quick visual indicator
on where things stand
 Examples:
 Vulnerabilities by
criticality/month
 Incidents by criticality/month
 Total block events (AV, FW,
Proxy, etc.)

46. VISUALIZATION TECHNIQUES: CLUSTERED STACKED GRAPH
Steps to create:
 Select your data, and create a
“stacked column chart” in Excel
 Make sure each data element is
setup as a “stacked column”
 To create a total for each
column, include this in your
source data, and enter it as a
“Line” under chart type. Give the
line a blank “fill” and “line” and
you’ll get the numbers only
 Be sure your data includes a
total row
For more, see: http://www.exceldashboardtemplates.com/how-to-easily-
create-a-stacked-clustered-column-chart-in-excel/

47. VISUALIZATION TECHNIQUES: THE SCORECARD
 Captures day-to-day operations in security
 One-page roll-up that can be presented to CIO, or used internally
 “Operations” section captures work being done: creating firewall rules,
patching systems, conducting awareness events
 The “Risk” section captures visibility into risk at the organization.
Number of scans, open vulnerabilities
 To the far right is the legend explaining the thresholds for each item

48. VISUALIZATION TECHNIQUES: THE SCORECARD