This document discusses the economics of penetration testing and introduces a new "next gen" model. Traditional penetration tests have hidden costs such as scoping, contracting, and remediation activities that extend beyond the testing period. They may also deliver poor value with low actual testing time and inability to validate remediation. The new model proposed offers an annual contract with on-demand testing, real-time results, direct access to analysts, and verification of remediation to reduce costs while improving security.

1. Economics of a Penetration Test
Simon Roe
January 2019

2. Agenda
2
• Penetration Testing – What is it and why
is it important
• The economics of Penetration testing
• ‘Next Gen’ Penetration testing – all the
cool kids are doing it
• Challenge your thinking – embrace the
21st century

3. Who are Outpost24
3
• Global HQ – Sweden
• Sales – BeNeLux, DACH,
Nordics, UK&I/France, US
• MSSP and Reseller partners
in additional locations
• Over 130 full time staff
• €11M+ turnover in 2018, up
37% in core markets

4. Penetration testing : What is it and why is it important?
4
A penetration test, colloquially known as a pen
test, is an authorized simulated cyber attack on a
computer system, performed to evaluate the
security of the system. The test is performed to
identify both weaknesses (also referred to as
vulnerabilities), including the potential for
unauthorized parties to gain access to the
system's features and data, as well as
strengths, enabling a full risk assessment to be
completed. - Source Wikipedia
Helps you understand your application or
networks ‘Attack Surface’

5. Should we be doing ‘Penetration testing?’
5
• Yes. The benefits gained are very valuable
• Find the backdoors – or things your automated tools miss
• Risk prioritisation
• Improve detection and alerting (OWASP top 10 2017 A10)
• Validate your controls
• Comply with local, national and international regulations
• It helps fill in the blanks from your automated scanning

6. How frequently should we do Penetration testing?
• How often does your application change?
• Is it a business critical application?
• Are you simply Pen testing for ‘tick box’ compliance or is it part of your DevSecOps
program?
Many customers pen test once or twice per year or quarterly as mandated by the
business or by a 3rd party
Poll: How many pen test(s) per application are you doing annually?

7. Economics of Pen testing…

8. This is what you think you pay for
A 10 day total Penetration test at an agreed ‘Day rate’ ($750-$1,000+)
Test Application (10 Days / $7,500)
$$

9. But you’ve forgotten your upfront costs
Appoint company, negotiate
contract (5 days / $2,500)Tender (2 days / $1000)
Scope, agree start date
(2 Days/ $1,000)
A 10 days Pen test really costs you 15 – 20
days by the time the test is finished
Upfront cost + Test Application (10 + 9 = 19 Days / $12,000)
+ $4,500
$$$$
+ 9 days
The day rate of your in house staff ($500)

10. And your ‘after test’ costs
Review the report
(3 days / $1,500)
Create remediation
issues ( 2 days /
$1,000)
A 10 day test, is likely 8 days testing, you have delays, issues and the report hand over
And then you have long road of unsure remediation ahead
Hand over & Review
(1 – 2 days / $500)
Remediate – (10+ day
/$5,000)
Upfront cost + Test Application (19 – 2 = 17 Days / but the
cost is still $12,000)
$$$$$$ Adding another 16 days and $8,000 =
potentially $20,000!!

11. $$
$$$$
$$$$$$
• Go to tender
• Find your supplier
• Scope out the app
• Negotiate the contract
• Review the findings
• Add them to your issue
tracking / backlog
• Remediate
The true cost of a pen test

12. Worse still…..
It delivers poor value
12

13. You don’t get what you pay for
• You have to be ready for the testers
• Delays happen you lose a day
• The production application is impacted
• Business complains. You have to restart. You lose a
half day and change the testing scope
• The tester needs to write a report and present the findings
to you
• That takes 1.5 / 2 days to deliver
• You 10 days of testing is likely 5 days of testing, a couple of
days of waiting, a day to write a report and a close out
meeting
• Your tester probably used 50%+ automation

14. You are left wondering
• You can’t validate your remediation efforts
• Maybe your automated scanner can help a little
• But DAST complements, not replaces manual pen test
• It wont learn your business logic
• It wont find that vulnerable plugin that gives Admin
access to your app
• It might run that Buffer overflow or SQLi attack –
damaging your app (or you might be black listing
those pages of the application, weakening its ability
to find risks)
• It won’t deliver zero false positives, therefore it might
actually increase the remediation activities

15. It costs you more money and delivers less value
• You think of the ‘test’ as a number of ‘Man days’. It’s a false
economy. You miss all the other costs before and after the test
• Your test is likely 50% automation with some review of findings,
a day for the reports and a day for the hand over
• You cannot work on remediation until the test finishes (delays)
and you have been given the report
• Likely you will be juggling false positives, subjective findings and
have no real way to query or clarify the issues
• You cannot easily verify that your development teams have
fixed the issues reported
And yet you still do it

16. “The definition of insanity is doing the
same thing over and over again,
but expecting different results.”
Albert Einstein
16

17. ‘Next Gen’ Penetration testing

19. What did our customers, your peers, tell us about the process?
A. They wanted a single contract, negotiated at the start of a 12 month
period to cover all testing.
B. They wanted to call off tests as needed – on demand.
C. They wanted to pay a fixed price for the pool or block of tests
D. They wanted it to be flexible. If they need more, they pay for more

20. What did our customers, your peers, tell us about service delivery?
A. They didn’t want to scope each application before hand
B. They wanted a flexible test, that is thorough and meets the purpose they want
C. They didn’t want to worry about ‘days’
D. They wanted the results to be presented in a single UI, with an option to integrate into
their CI/CD toolchain

21. What did our customers, your peers, tell us about remediation?
A. They wanted no false positives. If its presented in the UI it needs remediating
B. They didn’t want to wait until the end of the test and handover to start remediation
C. They wanted to be able to ask questions or seek clarification beyond the test length
D. They wanted to verify that their remediation efforts were successful, beyond the test
length
E. They wanted to fulfil the ‘check box’ compliance with a penetration test report

22. If you build it, they will come
• We listened. We built. Our customers came
• We delivered:
• Annual contracts
• Pool of testing, requested on demand
• Zero false positives
• Findings posted to UI available for remediation
during testing
• Direct access to the analysts
• Ability to request verification of remediation
activities
• Customers save time & money whilst being able to request
testing that suits their timelines and SDLC processes

23. Save yourself time and $$$ Now!

24. Old vs New
Old : hidden costs per test
New: fixed upfront cost
Old: You test when you can, and
pause your Dev.
New: Test on demand, as part of
the Dev process
Old: you remediate long after the
test has finished
New: you remediate whilst the test
is still ongoing
$$

25. Challenge the old Pen Test model and embrace the Netflix model
• Single contract, one signature, coverage
for 12 months
• Pool of ‘on demand’ tests
• Self service results available whilst tests
still ongoing (VIA UI)
• No false positives
• Access to the analysts, for questions and
clarification
• Remediation of findings on demand
• One hand to shake for a fixed price (no hidden
cost)
• On demand and flexible
• Instant access, no delays, you get the episode
as soon as it’s available
• Excellent streaming experience (no buffering)
• Great customer service and easy to get touch
with knowledgeable reps
• Play and Pause flexibility and satisfaction
guaranteed

26. “Outpost24 offers the unique combination of a
manually-tuned pen test with 24x7 monitoring.
Its ability to cover known vulnerabilities is a
real plus compared to pen tests that have to be
arranged on a regular basis.”
Application Manager, from a medium enterprise bank
26

27. Challenge your existing pen tests!
Outpost24.com