ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING                        PRESENTED BY:                       ...
Agenda!                   Agenda!•  Background!            •    Background!•  Approach!              •    Approach!•  Exam...
Agenda!                   Background!•  Background!            •    As Security Consultants we write reports!•  Approach!•...
Agenda!                   What do Reports Say?!•  Background!            •      2007 - 2011!•  Approach!              •   ...
Agenda!                   Approach!•  Background!            •    Metrics – definition!•  Approach!        •  Definition!   ...
Agenda!                   Useful?!•  Background!            •    Metrics can be misleading!•  Approach!        •  Example!...
Agenda!                   Useful?!•  Background!            •    Metrics are not always 100% useful!•  Approach!        • ...
Agenda!                    Approach!•  Background!             •    Why?: illustrate useful information!•  Approach!      ...
Agenda!                   Annual Distribution of Project (Days)!•  Background!•  Approach!•  Examples!•  Challenges with  ...
Agenda!                   SensePost Metrics Proposal!•  Background!            •    Metrics extracted from report data:!• ...
Agenda!                   SensePost Metrics in Action: Timelines!•  Background!            •    Useful?!•  Approach!•  Exa...
Agenda!                   SensePost Metrics in Action: Threat Metrics!•  Background!            •    Useful?!•  Approach!•...
Agenda!                   SensePost Metrics in Action: Bug Classes!•  Background!            •    Useful?!•  Approach!•  E...
Agenda!                   SensePost Metrics in Action: Top 10!•  Background!            •    Useful? !•  Approach!•  Examp...
Agenda!                   SensePost Metrics in Action: Re-Test!•  Background!            •    Useful?!•  Approach!•  Examp...
Agenda!                   SensePost Metrics in Action: Benchmarks!•  Background!            •    Useful?!•  Approach!•  Ex...
Agenda!                   Challenges!•  Background!            •    Bug counts vs bug classes!•  Approach!                ...
Agenda!                   Q&A!•  Background!            •    Thank you!•  Approach!•  Examples!              •    Longer p...
Upcoming SlideShare
Loading in...5
×

Application Assessment Metrics

549

Published on

Presentation by Yvette du Toit at ISSA in 2011.

This presentation is about application assessment metrics and their challenges. Examples of Sensepost metrics are given.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
549
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Application Assessment Metrics

  1. 1. ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING PRESENTED BY: Yvette du Toit
  2. 2. Agenda! Agenda!•  Background! •  Background!•  Approach! •  Approach!•  Examples!•  Challenges with •  Examples! Application Security Metrics! •  Challenges with Application Security Metrics!•  Q&A! •  Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  3. 3. Agenda! Background!•  Background! •  As Security Consultants we write reports!•  Approach!•  Examples! –  Test, analyse, write up findings, submit to client!•  Challenges with •  Issues still remain open – why?! Application Security Metrics! –  Reports not say enough!•  Q&A! –  Question value report offer! •  Solution – metrics / visualisation! –  Graphs, colour, size etc! •  First – letʼs take a look at what reports say…! –  Qualitative ratings! –  Best practice! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  4. 4. Agenda! What do Reports Say?!•  Background! •  2007 - 2011!•  Approach! •  Many words….!•  Examples!•  Challenges with •  Content (Exec Summary, Technical Summary, Conclusion)! Application Security Metrics! •  Are actions effective?!•  Q&A! •  What would be more valuable – comparison (time & peers)! •  How do we use metrics?! Pages Words Assessments 638 224587 Re-Tests 137 28164 Total 775 252751 ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  5. 5. Agenda! Approach!•  Background! •  Metrics – definition!•  Approach! •  Definition! –  Quantifiable!•  Examples! –  Characteristics!•  Challenges with Application Security •  3 Metric Veterans:! Metrics!•  Q&A! –  Jacquith - “those that support decision making about risk for the purpose of managing that risk” ! –  Marty – “a picture paints a thousand log records”! –  Godin: “just because something is easy to measure doesnʼt mean itʼs important”! •  NB: To measure what is important & that will yield “useful” information! –  Examples of metrics not necessarily useful! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  6. 6. Agenda! Useful?!•  Background! •  Metrics can be misleading!•  Approach! •  Example!•  Examples!•  Challenges with Application Security Metrics!•  Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  7. 7. Agenda! Useful?!•  Background! •  Metrics are not always 100% useful!•  Approach! •  Example!•  Examples!•  Challenges with Application Security Metrics!•  Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  8. 8. Agenda! Approach!•  Background! •  Why?: illustrate useful information!•  Approach! –  Recurring issues! •  Introduction!•  Examples! –  Time required to compromise!•  Challenges with –  Top 10 list! Application Security Metrics! –  Effectiveness of remediation!•  Q&A! –  Benchmarking! •  Who? 7 organisations in financial sector! •  When? 3 ½ years! •  How? Data capture process! –  Marco Slaviero (Head of R&D)! –  Spreadsheet for data capture! –  Report meta-data (project length, frameworks, dates etc.)! –  Findings categorised (pre-defined list of vulns)! –  Findings ranked (Impact, EoE, Threat metric)! •  Normalisation ! –  Allows for comparison across time and peers ! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  9. 9. Agenda! Annual Distribution of Project (Days)!•  Background!•  Approach!•  Examples!•  Challenges with Application Security Metrics!•  Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  10. 10. Agenda! SensePost Metrics Proposal!•  Background! •  Metrics extracted from report data:!•  Approach! •  Our Metrics! –  Timelines (plotting projects on timeline)!•  Examples! –  Basic counts and statistics (uncover counts)!•  Challenges with •  Number of projects! Application Security Metrics! •  Number of days!•  Q&A! •  Number of words and pages in report! –  Threat metrics (Findings per threat level)! –  Bug class metrics (Findings across categories) ! –  Top 10 list ! –  Re-Test Metrics! –  Benchmarks (comparison to peers)! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  11. 11. Agenda! SensePost Metrics in Action: Timelines!•  Background! •  Useful?!•  Approach!•  Examples!•  Challenges with Application Security Metrics!•  Q&A! ! "#$%&! ()*&! !"#$%&()*++%++#%,-+ ./0 112304 !"#$%&()5%67%+-+ 8/4 108.2 7(-9: 443 131438 ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  12. 12. Agenda! SensePost Metrics in Action: Threat Metrics!•  Background! •  Useful?!•  Approach!•  Examples!•  Challenges with Application Security Metrics!•  Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  13. 13. Agenda! SensePost Metrics in Action: Bug Classes!•  Background! •  Useful?!•  Approach!•  Examples! •  See 56% of findings occur in Top 11 bug classes!•  Challenges with Application Security •  2008 Anomaly (No Re-Tests) ! Metrics!•  Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  14. 14. Agenda! SensePost Metrics in Action: Top 10!•  Background! •  Useful? !•  Approach!•  Examples!•  Challenges with Application Security Metrics!•  Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  15. 15. Agenda! SensePost Metrics in Action: Re-Test!•  Background! •  Useful?!•  Approach!•  Examples! •  29% Critical and 42% High-risk issues remain open !•  Challenges with Application Security Metrics!•  Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  16. 16. Agenda! SensePost Metrics in Action: Benchmarks!•  Background! •  Useful?!•  Approach!•  Examples! •  Our client positioned 3rd (not highlighted here)!•  Challenges with Application Security Metrics!•  Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  17. 17. Agenda! Challenges!•  Background! •  Bug counts vs bug classes!•  Approach! –  Bug counts – number of findings!•  Examples!•  Challenges with –  Bug classes – categories! Application Security –  2 applications scenario (10 findings 1 bug class vs 1 finding in 10 bug classes)! Metrics!•  Q&A! •  Depth vs breadth! –  Each occurrence – depth! –  Each bug class - breadth! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  18. 18. Agenda! Q&A!•  Background! •  Thank you!•  Approach!•  Examples! •  Longer paper – mail me!•  Challenges with Application Security •  Email: yvette@sensepost.com! Metrics!•  Q&A! •  Contact: +27 79 509 8913! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×