Presentation on challenges facing the pen testing community and some possible solutions.

1. Failures in Penetration Testing
(and some strategies for success)
OSD IA Awareness Day
June 22, 2011
RISK, COMPLIANCE, AND SECURITY | ENTERPRISE RESILIENCE | INFORMATION TECHNOLOGY | PROGRAM & PROJECT MANAGEMENT | INTELLIGENCE

2. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Who we are
 Veris Group
 Management and technology services firm with a core focus
in information security.
|
INFORMATION TECHNOLOGY
 Clients include:
 Carnegie Mellon University (CMU), Software Engineering
Institute (SEI)
 Department of Justice
|
 Department of Homeland Security
ENTERPRISE RESILIENCE
 Department of the Treasury
 Department of Defense
 Social Security Administration
|
 Multiple commercial customers including Fortune 500
RISK, COMPLIANCE, AND SECURITY
companies
 Conduct penetration tests and develop penetration testing
programs for various Federal agencies.
2

3. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Who I am
 David McGuire
 Senior Security Engineer with Veris Group
|
INFORMATION TECHNOLOGY
 Lead penetration tester for multiple Federal and
commercial clients
 Previously, senior technical lead for NSA Red
Team
|
ENTERPRISE RESILIENCE
 Train a the Black Hat Technical Security Conference
 Vice-Chair of the NBISE Operational Security Tester
competency development panel
|
RISK, COMPLIANCE, AND SECURITY
3

4. What do all of these have in common?
4
RISK, COMPLIANCE, AND SECURITY | ENTERPRISE RESILIENCE | INFORMATION TECHNOLOGY | PROGRAM & PROJECT MANAGEMENT | INTELLIGENCE

5. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
What do all of these have in common?
 Relatively unsophisticated penetrations that are
conducted on a far larger scale than “nation state”
threats
|
INFORMATION TECHNOLOGY
|
ENTERPRISE RESILIENCE
|
RISK, COMPLIANCE, AND SECURITY

6. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Security Assessments Don’t Match
Real World Threats
Red Team
Technical difficulty of attack
Nation State Level Threats Assessments
|
INFORMATION TECHNOLOGY
Largest Percentage of
Threats
|
ENTERPRISE RESILIENCE
(Hacktavists,
Most Criminal Orgs, etc…)
|
Vulnerability
RISK, COMPLIANCE, AND SECURITY
Assessments
Unsophisticated Threats
Level of Effort, Cost and Timeframe of Assessment
6

7. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
What is a Penetration Test?
 One definition:
“A method of evaluating the security of a computer
|
INFORMATION TECHNOLOGY
system or network by simulating an attack from a
malicious source that may involve active exploitation of
security vulnerabilities. The process involves an active
analysis of the system for any potential vulnerabilities
that may result from poor or improper system
|
configuration, known and/or unknown hardware or
ENTERPRISE RESILIENCE
software flaws, or operational weaknesses in process or
technical countermeasures.”
|
RISK, COMPLIANCE, AND SECURITY
 Nice statement, but how do we define a
service out of it.
7

8. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
What is a Penetration Testing
Service?
 Finding vulnerabilities in applications and protocols through
custom exploit development?
A service where exploits and tools may need to be written
|

INFORMATION TECHNOLOGY
on the fly during the assessment
 Identifying and exploiting code and business logic
insecurities in web applications?
 Tricking someone into divulging sensitive information?
|
 Testing the physical security protections of an organization?
ENTERPRISE RESILIENCE
 Cracking a network perimeter, exfiltrating data and
demonstrating impact of successful penetrations
 Attempting to gain access to a system while evading
security monitoring capabilities?
|
RISK, COMPLIANCE, AND SECURITY
 Finding as many weaknesses in technical controls as quickly
as possible?
 Simply validating findings identified during a vulnerability
assessment?
8

9. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
What is a Penetration Testing
Service?
 Finding vulnerabilities in applications and protocols through
custom exploit development?
A service where exploits and tools may need to be written
|

INFORMATION TECHNOLOGY
on the fly during the assessment
 Identifying and exploiting code and business logic
insecurities in web applications?
 Tricking someone into divulging sensitive information?
|
 Testing the physical security protections of an organization?
ENTERPRISE RESILIENCE
 Cracking a network perimeter, exfiltrating data and
demonstrating impact of successful penetrations
 Attempting to gain access to a system while evading
security monitoring capabilities?
|
RISK, COMPLIANCE, AND SECURITY
 Finding as many weaknesses in technical controls as quickly
as possible?
 Simply validating findings identified during a vulnerability
assessment?
9

10. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
One Current Approach to “Definitions”
 We sometimes define a penetration tests by the
level of knowledge the tester will have of the
infrastructure to be tested:
|
INFORMATION TECHNOLOGY
 White Box: Full prior knowledge
 Black Box: No knowledge
 Grey Box: Some variation in between
|
ENTERPRISE RESILIENCE
|
RISK, COMPLIANCE, AND SECURITY
10

11. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
One Current Approach to “Definitions”
 We sometimes define a penetration tests by the
level of knowledge the tester will have of the
infrastructure to be tested:
|
INFORMATION TECHNOLOGY
 White Box: Full prior knowledge
 Black Box: No knowledge
 Grey Box: Some variation in between
|
ENTERPRISE RESILIENCE
 Extremely difficult to account for uncontrollable
factors.
 What benefit is derived by giving the tester no
|
knowledge of the system and only a week to
RISK, COMPLIANCE, AND SECURITY
test when a real attacker has months?
 An extremely poor differentiator as such
classification leads to far too much opacity and
un-measurability.
11

12. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Another Current Approach to
“Definitions”
 A somewhat better approach is to define a
penetration tests by the technology and/or
activity:
|
INFORMATION TECHNOLOGY
 Network, Wireless, Web Application, Social
Engineering. etc…
|
ENTERPRISE RESILIENCE
|
RISK, COMPLIANCE, AND SECURITY
12

13. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Another Current Approach to
“Definitions”
 A somewhat better approach is to define a
penetration tests by the technology and/or
activity:
|
INFORMATION TECHNOLOGY
 Network, Wireless, Web Application, Social
Engineering. etc…
|
ENTERPRISE RESILIENCE
 Level of Effort can not be defined
 Comprehensiveness and quality of assessment
can not be defined
|
 Does not have the ability to factor in time and
RISK, COMPLIANCE, AND SECURITY
cost
 How do managers know what to procure to
produce something of value?
13

14. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
The End Result
 An environment perfectly suited for
penetration tests of insignificant quality.
|
INFORMATION TECHNOLOGY
 Ill-informed managers often select
organizations with little penetration testing
capability, producing inconsistent results that
cannot be measured for comprehensiveness or
effectiveness.
|
ENTERPRISE RESILIENCE
 Low quality penetration tests rarely lead to
expected, or satisfactory, outcomes.
 Organizations still vulnerable to fairly
|
RISK, COMPLIANCE, AND SECURITY
unsophisticated attacks
14

15. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
What About Just Using the Best?
 Couldn’t we just use Red Teams (or their
commercial equivalents) to perform all
|
testing?
INFORMATION TECHNOLOGY
 Full scale “red team” assessments for all
penetration tests.
 Good in theory, however won’t work:
|
ENTERPRISE RESILIENCE
 Assumes “the best” have all the capabilities
necessary
 Still doesn’t account for measurability or
consistency
|
RISK, COMPLIANCE, AND SECURITY
 Far more assessments needed than can
be provided by this specific workforce
15

16. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Community Wide Efforts for
Improving Competency
 NBISE: Establishing competency models for
various experience levels of penetration
|
testers
INFORMATION TECHNOLOGY
 CREST: Provide a robust certification
mechanism for individuals and companies to
provide a quality level of assurance
|
 Currently only active in the U.K.
ENTERPRISE RESILIENCE
 PTES: Create a community driven standard
methodology for penetration testing
|
RISK, COMPLIANCE, AND SECURITY
16

17. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
An Argument for Better Definition
Penetration testing is dead, long live
penetration testing
|
INFORMATION TECHNOLOGY
 As defined today, penetration testing is almost
impossible to scope
|
ENTERPRISE RESILIENCE
 Any attempt to limit what a “penetration test”
is will be met with resistance
 All arguments have some merit
|
RISK, COMPLIANCE, AND SECURITY
 No way to categorize assessments based on
levels of sophistication of attacks
17

18. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
A New Definition for Penetration
Testing
 Define “Penetration Testing” as a set of
objectives and assessment perspective, not a
|
service to be performed
INFORMATION TECHNOLOGY
 Penetration testing cannot defined in any
measurable context
 Too many disparate activities exist under the
|
context of penetration testing
ENTERPRISE RESILIENCE
 Objectives and perspective of penetration
testing still important in assessing real risk of
system vulnerabilities and impact from
|
RISK, COMPLIANCE, AND SECURITY
successful compromise
 Define a set of security assessments designed
to meet the goals of penetration testing
18

19. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Penetration Testing Goals
 Independently assess a system from the
viewpoint of a malicious attacker, whether a
|
malicious insider or an uninformed outsider.
INFORMATION TECHNOLOGY
 Leverage organizational vulnerabilities (both
technical and non-technical) to assist in
determining business impact from a successful
attack.
|
ENTERPRISE RESILIENCE
 Test information security detection and
response capabilities in ways only an actual
cyber-attack can.
|
RISK, COMPLIANCE, AND SECURITY
 Test a system with active exploitation tools
and techniques, validating both technical and
non-technical vulnerabilities.
19

20. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Penetration Testing Goals
 Independently assess a system from the
viewpoint of a malicious attacker, whether a
|
malicious insider or an uninformed outsider.
INFORMATION TECHNOLOGY
 Leverage organizational vulnerabilities (both
technical and non-technical) to assist in
determining business impact from a successful
attack.
|
ENTERPRISE RESILIENCE
 Test information security detection and
response capabilities in ways only an actual
cyber-attack can.
|
RISK, COMPLIANCE, AND SECURITY
 Test a system with active exploitation tools
and techniques, validating both technical and
non-technical vulnerabilities.
20

21. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Better Defined Services
 Assessments should first be defined by “level
of effort,” objectives, and the required tester
|
skillset
INFORMATION TECHNOLOGY
 Goal to produce consistent, repeatable, and
measureable assessments that can follow a
recognized methodology
|
 Defined in a manner that is fair and
ENTERPRISE RESILIENCE
understandable to both penetration testers
and customers
|
RISK, COMPLIANCE, AND SECURITY
21

22. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Possible set of “Penetration Testing”
Services
 Operational Vulnerability Assessments
 the use of automated tools with limited
|
INFORMATION TECHNOLOGY
manual assessment techniques in order to
identify vulnerabilities within a system that
could lead to exploitation by an attacker
 While not a “penetration test,” often the first-
|
level of security assessment
ENTERPRISE RESILIENCE
 Often complementary to actual adversarial
testing conducted with limited resources
|
RISK, COMPLIANCE, AND SECURITY
22

23. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Possible set of “Penetration Testing”
Services
 Operational Security Assessments
 Combine vulnerability assessment and
|
INFORMATION TECHNOLOGY
penetration testing techniques, utilizing
available toolsets to conduct time and
resource limited adversarial assessments
 Provide the next level of independent security
|
assessments for security postures that have
ENTERPRISE RESILIENCE
matured beyond the need for vulnerability
assessments
 More on these later
|
RISK, COMPLIANCE, AND SECURITY
23

24. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Possible set of “Penetration Testing”
Services
 Advanced Organizational Security
Assessments
|
INFORMATION TECHNOLOGY
 A category of assessments defined as the
expert use of non-technical assessment
techniques to test organization personnel,
physical controls and procedural security
protections
|
ENTERPRISE RESILIENCE
 Operational Red Team Assessments
 Mimic as closely as possible a nation state or
highly-funded criminal organization cyber-
|
attack threat
RISK, COMPLIANCE, AND SECURITY
 Assessment timeframes would be measured
in multiple weeks or months
24

25. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Possible set of “Penetration Testing”
Services
 Advanced Proprietary Systems Security
Assessments
|
INFORMATION TECHNOLOGY
 A category of assessments specific to highly-
specialized implementations and extremely
technically in-depth
 The goal of these assessments is to provide
|
vendors and system implementers with the
ENTERPRISE RESILIENCE
real security risks associated with systems
not conducive to assessment with widely
available toolsets
|
 Probable other assessment types
RISK, COMPLIANCE, AND SECURITY
25

26. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Possible set of “Penetration Testing”
Services
 Advanced Proprietary Systems Security
Assessments
|
INFORMATION TECHNOLOGY
 A category of assessments specific to highly-
specialized implementations and extremely
technically in-depth
 The goal of these assessments is to provide
|
vendors and system implementers with the
ENTERPRISE RESILIENCE
real security risks associated with systems
not conducive to assessment with widely
available toolsets
|
 Probable other assessment types
RISK, COMPLIANCE, AND SECURITY
26

27. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Operational Security Assessments
 Fulfill the role of “penetration testing” today
for large, financially constrained organizations
|
that don’t have the ability or need to conduct
INFORMATION TECHNOLOGY
the full suite of adversarial activities
 Services that can be qualitatively measured,
with methodologies designed in a way that is
measurable and consistent
|
ENTERPRISE RESILIENCE
 Managers would not need to know the specific
technical details of the methodology as long
as it is known that a consistent set of
|
processes and procedures are followed
RISK, COMPLIANCE, AND SECURITY
27

28. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Operational Security Assessments
 Heavily focused on leveraging existing tools
vs. creation of tools during the assessment.
|
 Able to be consistently repeated across
INFORMATION TECHNOLOGY
disparate teams
 Targeted for the capabilities of mid-level team
members with senior-level team.
|
 Designed to be completed by a few testers in
ENTERPRISE RESILIENCE
limited timeframes
 Depending on the needs of the customer,
services could then be combined a-la-carte to
|
determine the overall engagement
RISK, COMPLIANCE, AND SECURITY
timeframes.
 Follow a semi-rigid methodology to ensure
consistency for assessments
28

29. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Operational Security Assessments
Sub-Services
 Operational Internal Network Security
Assessment
|
INFORMATION TECHNOLOGY
 Operational External Network Security
Assessment
 Operational Database Security Assessment
 Operational Web Application Security
|
ENTERPRISE RESILIENCE
Assessment
 Operational Wireless Security Assessment
 Operational Social Engineering Assessment
|
RISK, COMPLIANCE, AND SECURITY
29

30. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
A Hierarchy of Well Defined Security
Maturity of the system security Assessments
Operational Red Team
Nation State Level Threats
Technical difficulty of attack /
Assessments
|
INFORMATION TECHNOLOGY
Advance Proprietary
Largest Percentage of Threats System Assessments
(Hacktavists,
Most Criminal Orgs, etc…) Advanced Organizational
Security Assessments
|
ENTERPRISE RESILIENCE
Operational Security
Assessments
|
RISK, COMPLIANCE, AND SECURITY
Operational Vulnerability
Assessments Unsophisticated Threats
Level of Effort, Cost and Timeframe of Assessment
30

31. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Some Ways Forward
 Developing Methodologies
 The PTES is already moving in this direction,
|
just better define the categories of
INFORMATION TECHNOLOGY
assessments
 Gaining Acceptance
 Involvement of Community Leaders
|
ENTERPRISE RESILIENCE
 Independent Organizational Backing
 Governmental and regulatory endorsement
 Training to Specific Methodologies
|
RISK, COMPLIANCE, AND SECURITY
 We are already training to “Operational
Security Assessments”
 We are not enough, other training
organizations would need to also participate
31

32. INTELLIGENCE
PROGRAM & PROJECT MANAGEMENT |
Summary/Questions
 We are spending lots of time worrying about
nation-state level threats
|
 While important, the vast majority of attacks are
INFORMATION TECHNOLOGY
not a high level of sophistication
 These attacks have far larger scale than highly
technical attacks
 Red Team assessments will not be able to meet
|
ENTERPRISE RESILIENCE
the scale necessary
 Currently, “Penetration Testing” is defined in a
way that can adequately assess systems against
the vast majority of attacks
|
RISK, COMPLIANCE, AND SECURITY
 By defining a set of security assessments that
can follow a well defined methodology, we can
have some measure of assurance our systems
are protected against these “mid-level” threats
32