A practical walkthrough into why you need a manual pentest. Real reasons and metrics from the trenches. The presentation was delivered during an Episode of the SecRepo Podcast with hosts Mackenzie Jackson and Dwayne McDaniel, who asked very good questions.

1. Why should you do a pentest?
> Abraham Aranguren
> admin@7asecurity.com
> @7asecurity
> @7a_
+ 7asecurity.com
The Security Repo Podcast
2024-01-11
16:00 CET
1

2. Agenda
Why do you need a *manual* pentest?
→ Who am I
→ Intro to public pentest reports
→ 14 reasons why you need a *manual* pentest performed by *humans* :)
→ Other considerations:
• Shortcomings of automation
• Bug bounties
• Cheap “Pentests”
→ Case Study:
• What happens after multiple years of pentesting + fixing?
→ Q & A
2

3. → CEO at 7ASecurity, pentests & security training
public reports, presentations, etc.: https://7asecurity.com/publications
→ Co-Author of Mobile, Web and Desktop (Electron) app 7ASecurity courses:
https://7asecurity.com/training
→ Security Trainer at Blackhat USA, HITB, OWASP Global AppSec, LASCON,
44Con, HackFest, Nullcon, SEC-T, etc.
→ Founder and leader of OWASP OWTF, and OWASP flagship project: owtf.org
→ Some presentations: www.slideshare.net/abrahamaranguren/presentations
→ Some sec certs: CISSP, OSCP, GWEB, OSWP, CPTS, CEH, MCSE: Security,
MCSA: Security, Security+
→ Some dev certs: ZCE PHP 5, ZCE PHP 4, Oracle PL/SQL Developer Certified
Associate, MySQL 5 CMDev, MCTS SQL Server 2005
About Abraham Aranguren

4. Public Mobile Pentest Reports 2022-2023
Free & Fast way to learn about security = Read public pentest reports! :)
Download from: https://7asecurity.com/publications
2023 Public Pentest Reports:
→ Pentest-Report K-9 Mail, Fuzzing, Threat Model & Supply Chain Audit (OSTIF) 04.2023
→ Pentest-Report ArgoVPN Mobile, Servers & Privacy (OTF) 03.2023
→ Pentest-Report Bridgefy Web & Mobile apps, Cloud & Privacy Audit (OTF) 02.2023
2022 Public Pentest Reports:
→ Pentest-Report minivpn Go client & Desktop Apps (OTF) 08.2022
→ Pentest-Report Amnezia VPN Mobile & Desktop Apps (OTF) 07.2022
→ Pentest-Report Linux Foundation LFX Platform (OSTIF) 06.2022 (possibly in 2023)
→ Pentest-Report LeaveHomeSafe Mobile Apps (OTF) 04.2022
• COVID19 contact-tracing app enforced in Hong-Kong
→ Pentest-Report WEPN Web, API, Mobile & Device (OTF) 03.2022
4

5. Older Public Mobile Pentest Reports - I
Smart Sheriff mobile app mandated by the South Korean government:
Public Pentest Reports:
→ Smart Sheriff: Round #1 - https://7asecurity.com/reports/pentest-report_smartsheriff.pdf
→ Smart Sheriff: Round #2 - https://7asecurity.com/reports/pentest-report_smartsheriff-2.pdf
Presentation:“Smart Sheriff, Dumb Idea, the wild west of government assisted parenting”
Slides:https://www.slideshare.net/abrahamaranguren/smart-sheriff-dumb-idea-the....
Video: https://www.youtube.com/watch?v=AbGX67CuVBQ
Chinese Police Apps Pentest Reports:
→ "BXAQ" (OTF) 03.2019 - https://7asecurity.com/reports/analysis-report_bxaq.pdf
→ "IJOP" (HRW) 12.2018 - https://7asecurity.com/reports/analysis-report_ijop.pdf
→ "Study the Great Nation" 09.2019 - https://7asecurity.com/reports/analysis-report_sgn.pdf
Presentation: “Chinese Police and CloudPets”
Slides: https://www.slideshare.net/abrahamaranguren/chinese-police-and-cloud-pets
Video: https://www.youtube.com/watch?v=kuJJ1Jjwn50
5

6. Other pentest reports:
→ imToken Wallet - https://7asecurity.com/reports/pentest-report_imtoken.pdf
→ Whistler Apps - https://7asecurity.com/reports/pentest-report_whistler.pdf
→ Psiphon - https://7asecurity.com/reports/pentest-report_psiphon.pdf
→ Briar - https://7asecurity.com/reports/pentest-report_briar.pdf
→ Padlock - https://7asecurity.com/reports/pentest-report_padlock.pdf
→ Peerio - https://7asecurity.com/reports/pentest-report_peerio.pdf
→ OpenKeyChain - https://7asecurity.com/reports/pentest-report_openkeychain.pdf
→ F-Droid / Baazar - https://7asecurity.com/reports/pentest-report_fdroid.pdf
→ Onion Browser - https://7asecurity.com/reports/pentest-report_onion-browser.pdf
More here:
https://7asecurity.com/publications
Older Public Mobile Pentest Reports - II
6

7. 14 Reasons Why You Need A Manual Pentest
7

8. Introduction #1
Many people believe automated security tools can completely protect software (!).
This benefits:
1. Vendors: To sell ineffective products & services.
2. Cybercriminals: To exploit these issues for fun & profit.
Security pros know automated tools have flaws:
1. False positives: Waste your time & money as your staff reads & tries to
understand & mitigate bullshit findings.
2. False negatives: Leave your systems wide-open to existing vulnerabilities
automated tools failed to find.
8

9. Introduction #2
Just think about it:
If automation was enough …
… why do large companies like Google and Facebook use the following on top of
automation?
1. Huge in-house security teams
2. Hire pentests performed by external companies
3. Implement bug bounty programs on top of 1 + 2.
TLDR;
Automation can help, but it is not sufficient
9

10. 14 Reasons Why You Need A Manual Pentest
10

11. #1 Vulnerabilities Hiding Behind Complexity
11
Most dynamic automated tools will completely fail to reach vulnerable endpoints in the
following scenarios:
1. Date is only valid if the user is > 18 years old
2. Invalid parameter or parameter combination (i.e. Spanish postcode for UK address)
3. Required multi-step sequences prior to vulnerable endpoint
4. The tool fails to detect the user is logged out (i.e. session is invalid)
5. The tool triggers throttling/blocking mechanisms = Every request after that is
ignored.
etc.
What about static analysis tools?
They fare really poorly when frameworks & apps generate code on the fly:
Dynamically generated code will likely be completely missed.

12. #2 Logic Flaws
12
Logic flaws are pretty much:
● Impossible to find for both static and dynamic analysis tools
Example:
Raja Sekar Durairaj, was able to identify a logic flaw for which he was awarded a bounty
of $10,000 by Facebook 1 . The vulnerability was able to get your Facebook private
friend list, by registering a new Facebook account using the victim’s phone number and
then navigating to “Update Contact Info”, instead of confirming the SMS code.
https://medium.com/@rajsek/how-i-was-able-to-get-your-facebook-private-friend-list-resp
onsible-disclosure-91984606e682

13. #3 Information Disclosure
13
Information leaks are extremely difficult for automated tools to detect:
● Humans can easily see when “this data should NOT be readable!”
● Tools struggle with this…i.e. Probing for XSS → you get instead the full list of users
→ tool says “all OK, no XSS” xD
Example:
Dzmitry Lukyanenka discovered a vulnerability on Facebook. The bug allowed him to
read random server memory uploading a crafted GIF image. This is a type of information
disclosure bug for which he gained a bounty amount of $10,000.
https://www.vulnano.com/2019/03/facebook-messenger-server-random-memory.html

14. #4 Authorization Flaws
14
Should user A really be able to see or access X?
● Humans can answer that
● Tools can’t …
Example:
Philippe Harewood found a Facebook authorization flaw and a logic flaw for
which he received a sum total of $27.500 5 . The bug allowed attackers to
add themselves as an admin to any business, hence taking over any business
account and gaining access to various business assets (Facebook pages, Ad
accounts, applications, Instagram accounts) connected to the business.
https://philippeharewood.com/facebook-business-takeover/

15. #5 Business Logic Errors
15
Each business is unique and must conform to business requirements.
Basic example: Will a negative price purchase = a refund?
● Humans can try to bypass business logic
● Tools can’t …
Example:
Richard FitzGerald won a bounty of $1,000 for identifying a vulnerability which had the
potential to abuse pricing errors in saved carts in Shopify 7 . All Shopify stores not using
automated abandoned cart emails were susceptible to this vulnerability.
https://hackerone.com/reports/336131

16. #6 Subtle Injection Attacks
16
Injection attacks confuse attacker-supplied-data with instructions.
Many types: Code Injection, Command Injection, Cross-Site Scripting(XSS), email
header injection, SQL Injection (SQLi), NOSQL Injection, XML Injection, etc.
Static analysis automation can find some forms of injection, but not always, particularly
not when the vulnerable code is dynamically generated.
Example:
Frans Rosén won a bug bounty amount of $10,000 when he identified a command
injection vulnerability on SEMrush 16 . This was a Remote Code Execution vulnerability
on www.semrush.com/my_reports via a Logo upload.
https://hackerone.com/reports/403417

17. #7 API Implementation Flaws
17
APIs are extremely hard for automated tools to test because:
1. Trial & error is often required to invoke all endpoints properly
○ Can’t invoke? → Missed vuln
2. Humans can figure out required business logic & parameter combinations to invoke
endpoints
○ BUT tools can’t
Example:
Artem Moskowsky identified an exploit in Valve’s developer portal for reporting, he was
awarded a bounty of $20,000 17 . Moskowsky changed the parameters in the API
request to get codes for virtually any game regardless of ownership. People with a
developer account could generate as many keys as they wished too for any game hosted
on Steam. Rogue infiltrators could give away or sold off the activation codes and exploit
the vulnerability.
https://www.techspot.com/news/77402-valve-awards-20000-bug-bounty-exploit-...

18. #8 Remote Code Execution
18
Surely, automated tools will catch all Remote Code Execution flaws, right?
No, sometimes these are subtle and easy to miss, look at this:
Example:
United Airlines paid a bug bounty of 1.5 million miles to bounty hunter, Jordan Wiens from
Florida who reported two remote code execution bugs.
https://www.theregister.co.uk/2015/07/16/united_airlines_bug_bounty_18m/

19. #9 Low-Level Vulnerabilities
19
Static Analysis tools looking at the code fail to find low-level vulnerabilities such as:
1. Vulnerabilities in processors
2. Vulnerabilities in compilers
3. Vulnerabilities in subtle interactions between libraries used
4. Vulnerabilities in subtle interactions between app & third party components
5. Side-channel data leaks
etc.
Example:
Carl Waldspurger and Vladimir Kiriansky discovered two vulnerabilities which were
variants of Spectre Variant One and won a payout of $100,000. Spectre is a security
vulnerability which affects microprocessor chips. The first subvariant which was Spectre
1.1 would allow attackers to execute malicious code by exploiting a buffer overflow. In the
case of the second, Spectre 1.2 would allow attackers to overwrite read-only data and
manipulate the target computer.
https://www.securityweek.com/intel-pays-100000-bounty-new-spectre-variants

20. #10 Insecure Direct Object References (IDOR)
20
Should user A see data for user B?
● Humans can figure this out
● Tools struggle…
● And .. WAFs cannot stop ID=1 vs. ID=2
Example:
An insecure direct object reference vulnerability was reported in Australia Post’s “Click
and Send” online service as it facilitated users to expose the details of others by
changing a shipping ID number that appeared in the URL of a completed transaction. The
service was temporarily suspended by the company, on the grounds of a “system error”.
https://www.itnews.com.au/news/australia-post-customers-exposed-in-direct-object-refer
ence-flaw-317651

21. #11 Cross Site Scripting (XSS)
21
Surely, nowadays, automated tools will always catch XSS?
Not always, and especially not in these cases:
1. DOM-based XSS
2. XSS that involves encoding/decoding payloads
3. XSS that involves interactions between multiple websites
4. XSS from other edge cases
Example:
Thomas DeVoss identified a Cross Site Scripting (XSS) vulnerability on Mapbox & Firefox
which earned him $1,000. In 2016, he reported a reflected cross-site scripting issue in the
map embed page of v4 map API that affected Firefox users singularly. To resolve the
issue they switched to HTML-escaped underscore templates(<%-).
https://hackerone.com/reports/135217

22. #12 Server Side Request Forgery (SSRF)
22
Automated tools will often miss SSRF issues due to logic, complexity, etc.
But, even if they find it, they will never match humans to increase impact:
1. Exactly what can you do with this SSRF vulnerability?
2. What data/systems can you access?
3. Can attackers fetch cloud credentials or session data?
Example:
Sergey Toshin, reported a SSRF vulnerability to PayPal and won a bounty amount of
$10,000. A malicious attacker could supply a crafted URL to the Venmo application and
leak session data to an attacker-controlled website
https://hackerone.com/reports/401940

23. #13 Memory Corruption Vulnerabilities
23
Automated tool limitations:
1. Dynamic analysis will (at best) crash the app, but fail to explain why it happened
2. Static analysis will (at best) find usage of insecure functions, but fail to prove the
issue with an exploit that actually works
3. If the vuln is on a package used by the app, even static analysis will miss it :)
Example:
Vanhoecke Vinnie won a bug bounty of $18000 for a buffer overflow. In Steam and other
valve games (CGSO, TF2 and others) there is a functionality to seek game servers called
the server browser. They identified and reported a stack-based buffer overflow.
https://hackerone.com/reports/470520

24. #14 Multiple Flaws and Chained Vulnerabilities
24
Automated tools will never chain multiple vulnerabilities to increase impact, as attackers
will…
Example:
Mohamed M. Fouad revealed several critical vulnerabilities in the Starbucks website. The
vulnerabilities he identified included: Remote Code Execution, Remote File Inclusion lead
to Phishing Attacks and CSRF (Cross Site Request Forgery). These vulnerabilities would
enable cyber criminals to hijack customer accounts, collect credit card details and misuse
information.
https://www.adaware.com/blog/cream-sugar-and-security-bugs-another-starbucks-vulner
ability
https://mohamedmfouad.blogspot.com/2015/09/starbucks-critical-flaws-allow-hackers.ht
ml?view=classic
https://thehackernews.com/2015/09/hacking-starbukcs-password.html

25. 25
Other Considerations
Shortcomings of Automation, Bug bounties & Cheap “Pentests”

26. Shortcomings Of Automation #1
26
● Automated tools have Limited Understanding of Context:
○ Lack of ability to interpret contextual nuances
○ = Misinterpretation of potential vulnerabilities.
● Automated tools are unable to Mimic Human Intuition:
○ Lack of intuition and critical thinking vs. human testers
○ = Prone to misjudging some scenarios.
● Automated tools struggles with Novel Threats:
○ Reliance on predefined patterns
○ = Unable to find emerging or custom uncataloged threats.

27. Shortcomings Of Automation #2
27
● Automated Tools Overlook System-Specific Configurations:
○ Unable to adapt to unique setups
○ = Miss vulnerabilities specific to an org / app
● Automated Tools are Ineffective in Complex Environments:
○ Unable to navigate intricate systems
○ = False positives & False negatives.

28. Shortcomings Of Bug Bounty Programs #1
28
● Huge influx of invalid and fake submissions:
○ Lots of false alarms, duplicates & non-exploitable issues.
○ = Wasted time & effort to review and verify.
● Elevated workload for development teams:
○ Sorting through the high volume of bug reports
○ = Slow down progress on development tasks.
● Increased resource allocation for review and verification:
○ Validating lots of bug reports
○ = lots of resources, tools & technologies to check reports
○ = high cost

29. Shortcomings Of Bug Bounty Programs #2
29
● Reduced efficiency due to high noise-to-signal ratio:
○ Lots of BS submissions vs. Few useful/valid reports
○ = Valuable reports get buried in the noise + can be missed!
● Escalated overall program costs:
○ The combination of increased workload, resource allocation, and
reduced efficiency due to high noise-to-signal ratio
○ = Higher overall costs.

30. Shortcomings Of Bug Bounty Programs #1
30
● Huge influx of invalid and fake submissions:
○ Lots of false alarms, duplicates & non-exploitable issues.
○ = Wasted time & effort to review and verify.
● Elevated workload for development teams:
○ Sorting through the high volume of bug reports
○ = Slow down progress on development tasks.
● Increased resource allocation for review and verification:
○ Validating lots of bug reports
○ = lots of resources, tools & technologies to check reports
○ = high cost

31. Shortcomings Of Cheap “Penetration Tests”
31

32. Shortcomings Of Cheap “Penetration Tests” - Intro
32
If you were going to have heart surgery….
Would you:
Option 1) Choose the cheapest surgeon
OR
Option 2) Choose the best surgeon you can find
? :)

33. Shortcomings Of Cheap “Penetration Tests” 101
33
● Often copy-paste the output of automated tools
○ = NOT a pentest, false positives, false negatives, etc.
● Often use less skilled professionals:
○ = guaranteed false positives + false negatives
● Increased Business risk:
○ Cheap pentest = Missed low-hanging fruit vulnerabilities
○ = Loss of customer trust & brand damage
○ = Possible regulatory penalties
○ = You will likely get hacked :)

34. Case Study - Intro
34
TLDR; Why should I do a pentest?
… Show me the data?

35. Case Study: 1st Pentest Iteration
35
First Pentest results
3 directly exploitable vulnerabilities:
Identified Vulnerabilities:
1. XXX-01-003: Possible Phishing via HTMLi on Company Name (Medium)
2. XXX-01-008: Possible Phishing via Open Redirect on Cluster Logout (Low)
3. XXX-01-010: Possible MitM via Usage of Invalid Cluster Certificates (High)

36. Case Study: 2nd Pentest Iteration
36
Second Pentest Results
2 directly exploitable vulnerabilities
Identified Vulnerabilities:
1. XXX-02-001: Possible Phishing via Open Redirect on Cluster Login (Low)
2. XXX-02-007: RCE in ExternalKeyValidator via crafted SSH Key (Critical)

37. Case Study: 3rd Pentest Iteration
37
Third Pentest Results
0 directly exploitable vulnerabilities
Identified Vulnerabilities:
none

38. Case Study: Iteration Summary
38
Pentest iteration:
1. 3 directly exploitable vulnerabilities
2. 2 directly exploitable vulnerabilities
3. 0 directly exploitable vulnerabilities
Is regular pentesting valuable? (i.e. ~once / year)
Any pentester will tell you:
● 100% Yes!
● It will become increasingly hard to find anything in an app that is regularly
pentested + patched.

39. Final Thoughts
39

40. Final Thoughts
40
● Affordable “pentests” rely on automated tools & miss high/critical issues.
● Real test cost includes:
1. Consequences from missed vulns (= false negatives)
2. False alarms (= false positives = wasted effort)
● Skilled testers provide accurate insights, true findings and prioritize
remediation efforts effectively.
● Comprehensive tests drive proactive security strategies that work.
● Investing in quality audits improves defenses and ensures resilient
operations.
Manual testing is vital for uncovering complex vulnerabilities missed by
automation.
Automation complements but does NOT replace expert security audits.

41. Questions
41

42. > admin@7asecurity.com
> @7asecurity
> @7a_
> @owtfp [ OWASP OWTF - owtf.org ]
+ 7asecurity.com
Q & A
Free Pentest Contest 2023:
https://7asecurity.com/blog/2024/01/free-pentest-contest-2023-deadline-approa
ching/
1000 off your next pentest → code: SECREPO1000
● sales@7asecurity.com / https://7asecurity.com/#contact
● Public pentest reports → https://7asecurity.com/publications
40% off any training course → code: SECREPO40
● https://store.7asecurity.com/discount/SECREPO40
● Free workshops → https://7asecurity.com/free
42

43. Inadequate or limited reporting
43
After a penetration test, the report is a crucial deliverable providing insights and
recommendations for cybersecurity efforts. Cheaper services tend to fall short in
reporting:
● Superficial summaries
● Absence of contextual insights
● Generic recommendations
● Missing prioritization
● Limited post-test engagement

44. Prioritizing Quality Penetration Testing: A Brief Overview
44
● Thorough Analysis: Goes beyond surface-level examination, uncovering
concealed vulnerabilities.
● Customized Techniques: Tailored to specific infrastructure and operational
nuances for meticulous assessment.
● Expertise Matters: Seasoned professionals bring strategic insight,
complementing automated tools.
● Ongoing Engagement: Reputable services offer post-testing support,
addressing evolving threats and ensuring continued security.
● Comprehensive Reporting: Provides in-depth insights, impact assessments,
and prioritized remediation steps.