This document discusses the differences between penetration testing and vulnerability assessments, and why clients often receive poor quality tests. It notes that penetration tests are meant to be deeply interactive, focusing on achieving specific goals through exploitation, while vulnerability assessments only superficially identify issues. It also explains that clients contribute to poor tests by lacking understanding of the purpose and proper scope of each method, and not performing adequate quality control of testers. The document provides recommendations for how clients can improve tests by learning testing standards, clearly defining objectives, and incentivizing testers to achieve goals through payment structures.

1. Vlad Styran
Next Generation Pentest
Your Company Cannot Buy
why both consultants and customers
are doing it
wrong

2. Who’s that guy?
• Security Consultant for BMS Consulting
• Social Engineering researcher
• InfoSec blogger
• Podcaster

3. Why he is here?
• Pentesting since 2006
– Web sites, banking systems, telecom,
• X commercial pre-sale presentations
– Saw client’s eyes BEFORE the test
• X-Y pentest reports written
– Saw client’s eyes AFTER the test
– Writing reports is HELL
• Z pentest reports read
– Reading others’ reports is FUN
• CISSP, CEH, CISA…
– Because it rarely matters

4. Why are YOU here?
• This preso is for those who want a great
pentest to be done
– and someone to benefit from this pentest
• usually it’s a company
• You may be a customer
• Or a consultant
• Or both
• And you should agree that there’s something
wrong with pentesting industry

5. Some definitions
• What is a Penetration Test?
• What is a Vulnerability Assessment?
• What is the difference?
• Why should anyone bother?
• And let’s make it quick and simple

6. Test
• Testing is deeply interactive
• A test is something a tester and what is tested
do both
– We act and see the reaction
– Not just look, measure and record
– We touch, push and kick
– We challenge what we test
• Test has a goal

7. Penetration Test
• Penetration is getting through obstacles:
– Security systems
– User awareness
– Physical barriers
• The pentest succeeds if we get through
– And fails if we don’t
• And this usually means right the opposite to client
• The goal is virtually anything, but
– Penetrate a system
– Pwnz0r everything: DBA, root, Domain Admin
– ‘Get’ the data to show it’s vulnerable
– Show that the business might be stopped

8. Vulnerability Assessment
• Find all vulnerabilities
– Remove false positives (optional)
• And tell us how to fix them
– Usually in couple of deferent ways
• Don’t try to break anything, it might… break!
• Come in few weeks (months?) and check how
whether we fixed stuff

9. The Difference
• Deep interactivity:
– Pentest is interactive to the very deep you can get
– Vulnerability Assessment is superficial
• The goal:
– Pentest aims at a narrow goal
– Vuln Assessment is as broad as client can pay for
• The PenTest is focused and thorough
• The VA is a mile broad and a feet deep
• You can easily do VA yourself but PT isn’t easy
– Not because it’s hard to do, because of conflict of interest

10. More Difference
• PT not just scans, it exploits
• Most pentest standards do multiple channels
– Systems and network
– Wireless and telecom
– Human interaction
– Physical stuff
• VA is purely technical
– Systems and network
– And maybe wireless… or telecom…

11. That was ‘what’ and ‘how’.
What about ‘why’?
• And this is the most important and interesting
part that everyone should know
• Vulnerability Assessment:
“Let us know how we can fix what is presumably
already broken”
• Penetration Test:
“Try to break what is presumably unbreakable”*
*Considering reasonable time and resources available

12. Now To Work
• Why clients buy pentests?
• How consultants do pentests?
• Why clients get bad pentests?
• What can we do to fix it?
– Clients
– Pentesters

13. How consultants do pentests?
• We set the scope
– Systems, locations, people, contacts etc.
• We do recon
– Short for ‘reconnaissance’
• We enumerate the targets
– And search for vulnerabilities
• It is pretty much the VA until this point

14. How consultants do good pentests?
• We validate the vulnerabilities
– ‘Validate’ stand for ‘exploit’ since business people
don’t like hacker jargon
• We leverage access gained and pivot further
– Into the network, into the sun, into the cookies…
• We collect evidence of your data compromise
– Without actually compromising the data
– But enough to make your bosses like OMG

15. How consultants do outstanding
Pentest-NG?
• We meet your business people beforehand
– To know how your business lives
– And research on how someone can kill it
• We do all channels and vectors
– We plan for HR interviews and local conferences
– We write custom software and exploit code
• We do virtually anything to make you cry over
your spent InfoSec dollars

16. Why clients buy pentests?
• Want to test the security
– The only true reason which is really rare
• Compliance
– That mandates pentests
• Want to know the risks
– Although there are much better and safer tools
• False compliance
– That does not mandate pentests
• We were hacked!!
• Have no idea how else to ‘fix it’…

17. Why clients get/do bad pentests?
What clients cannot affect
• Bad pentesters
– Some pentesters just suck
• Most methodologies suck too
– Remember your high school lessons
• Time/cost relation in consulting business
models
– Pentests are quick
– ‘Quick’ means ‘cheap’

18. Why people get/do bad pentests?
Clients can and do affect
• Lack of understanding the difference
– Most buy a plain VA dressed as a sexy pentest
• Lack of understanding the reason
– PCI pentest not to find vulns, you have ASV scan
for that
• Lack of quality assurance
– It takes to buy 2-3 bad pentest to understand
they’re bad
• Validation panic

19. How to clean this s fix this
• Learn and understand the difference
– Read the PT standards – there are plenty
• PTES, OSSTMM, NIST, ISACA, ETC.
• Reason which are good for you
– Ask pentesters you know are really good
• Twitter, mailing lists, security conferences…
• Learn and understand the reason
– Define why are you doing this before posting a PO
– Reason about it and choose the best you need
• PT or VA

20. How else can we fix this?
• Change the payment rules
– Create the list of objectives
– Pay a ‘standard’ price for reformatted Qualys
report Vulnerability Assessment
– Pay a bonus for each objective in the list
• Choose good pentesters
– Ask for papers (sample reports, certs, references)
• NDA excuse is bull s irrelevant
– Arrange demo exercises
• (Good) pentesters love exercises
• Honeypots are for free

21. How else can we fix this?
(dirty tricks)
• Have nerve
– Stress on the need of PT over VA or vice versa –
based on your need
• Push on compliance
– PCI Information Supplement 11.3
• Requires the vulns to be exploited
• Requires channel diversity: social, network, WiFi etc.
• Learn some skill yourself
– It really helps
– And it’s really fun

22. Something to Think About and Discuss
• Vuln Assessment covers a small portion of preventive controls
• PenTest delves into each and every control you have
• Assume you have no need in testing
preventive controls… Just assume
• How can you test reactive and
corrective controls?

23. Thank you… in advance!
vlad@styran.com
https://secureglaxy.blogspot.com
@saprand