This document discusses continuous API security testing using OWASP ZAP on Kubernetes. It proposes running ZAP scans against microservices continuously or on-demand directly in the Kubernetes cluster. This would find vulnerabilities earlier in the development process compared to only running security tests at the end. The document outlines how Testkube, a Kubernetes-native testing framework, could be used to orchestrate and run ZAP scans as tests against microservices to achieve continuous security testing. Demo projects are referenced to show how this approach could work.

1. qaware.de
Kontinuerliche Sicherheitstests für APIs
mit Testkube und OWASP ZAP
Mario-Leander Reimer
mario-leander.reimer@qaware.de
@LeanderReimer
@heise_devSec #devSec23 @testkube_io

2. 2
Mario-Leander Reimer
Managing Director | CTO
@LeanderReimer
#cloudnativenerd #qaware
#gernperDude

3. "Software Is Eating the World."
Marc Andreessen, 20th August 2011

4. Holistic security still seems to be an often
neglected non-functional requirement in many
software projects and agile teams.

5. Security is one of several software product quality attributes.
Which one is more important?
QAware | 5
Software Product
Quality
(ISO 25010)
● Modularity
● Reusability
● Analysability
● Modifiability
● Testability
Maintainability
● Confidentiality
● Integrity
● Non-repudiation
● Authenticity
● Accountability
Security
● Adaptability
● Installability
● Replaceability
Portability
● Co-existence
● Interoperability
Compatibility
● Maturity
● Availability
● Fault Tolerance
● Recoverability
Reliability
● Time Behaviour
● Resource Utilization
● Capacity
Efficiency
● Completeness
● Correctness
● Appropriateness
Functional Suitability
● Operability
● Learnability
● UI Aesthetics
● Accessibility
Usability
Deployability
Safety

6. QAware | 6
Monolithic systems were relatively
easy to test.
■ No distribution, no IPC
■ Homogene technology stack
■ Low infrastructure complexity
■ Managed infrastructure
■ Long release and test cycles
■ Developed by one team

7. QAware | 7
Microservice-based systems are complex.
Testing them is even more complex.
■ High distribution with various communication channels and IPC formats
■ Heterogeneous Technology Stacks
■ High infrastructure complexity with many components
■ New operating model with more responsibility for the developers
■ Short release cycles. Many teams.

8. All modern IPC protocols are susceptible to attacks from the
OWASP API Security Top 10
QAware | 8
GraphQL
gRPC
REST

9. All modern IPC protocols are susceptible to attacks from the
OWASP API Security Top 10
QAware | 9
GraphQL
gRPC
REST
API1:2023 Broken Object Level Authorization
API2:2023 Broken Authentication
API3:2023 Broken Object Property Level Authorization
API4:2023 Unrestricted Resource Consumption
API5:2023 Broken Function Level Authorization
API6:2023 Unrestricted Access to Sensitive Business Flows
API7:2023 Server Side Request Forgery
API8:2023 Security Misconfiguration
API9:2023 Improper Inventory Management
API10:2023 Unsafe Consumption of APIs
A01 Broken Access Control
A02 Cryptographic Failures
A03 Injection
A04 Insecure Design
A05 Security Misconfiguration
A06 Vulnerable and Outdated Components
A07 Identification and Authentication Failures
A08 Software and Data Integrity Failures
A09 Security Logging and Monitoring Failures
A10 Server Side Request Forgery (SSRF)

10. Mastering the tools, techniques and technologies required for
Continuous Delivery is not easy!
QAware | 10
Continuous
Delivery
Low Risk
Releases
Less
Rework
Fast Time
to Market
Better
Products
Lower
Costs
Happier
Teams
Happier
Users
Loosely Coupled Architectures
Maintainable Code
Empowered Teams
Continuous Security from Day 1
Test
Automation
Continuous
Integration
GitOps
Deployment
Automation
Monitoring and Alerting

11. OWASP Zed Attack Proxy (ZAP)
QAware | 11
■ Widespread and well-known open source web application vulnerability scanner
■ Detailed documentation. International community.
■ Several modes of operation: Intercepting Proxy, Active und Passive scanner, HTTP Spider, Brute
Force Scanner, Port Scanner, OpenAPI v3, SOAP, GraphQL, Web Sockets
■ ZAP provides a powerful API and tools for Security Scanning Automation
■ The official ZAP Docker images provide an easy way to run ZAP, especially in CI/CD and container
runtime environments such as Kubernetes
– API Scan - a full scan of an API defined using OpenAPI / Swagger, or GraphQL
– Baseline Scan - a time limited spider which reports issues found passively
– Full Scan - a full spider, optional ajax scan and active scan which reports issues found
– Webswing - run the ZAP Desktop UI in a browser
■ GitHub Action available for easy integration into GH build pipelines
■ https://www.zaproxy.org/docs/

12. Monolithic, linear CI/CD pipelines are suboptimal and will
result in delayed feedback and long release cycles.
QAware | 12
Usually delayed until the end of sprint or the release.
Which one first? Functionality vs. Performance vs. Security?

14. A microservice architecture with many downstream
dependencies is complex and really hard to test.
QAware | 14
Cluster
Microservice
A
Microservice
B
Microservice
C
External
System X
External
System Y
Team A Team C Team B Unknown

15. Why not run (non)-functional tests against a
cloud-native microservice architecture
continuously, or triggered on the cluster itself?

16. Initial idea and conceptual architecture for continuous API
security tests with ZAP on Kubernetes
QAware | 16
default zap
Security
Unit Test
Tester
Microservice
Deployment
API Test
ZAP API
ZAP GUI
REST
CronJob
HTML
Pod
Pod

17. Improved Conceptual Architecture
QAware | 17
Packages
Package
publish
update
Run
deploy
watch
Deploy
watch
Dev GitOps
Build
push
Checkout Build Test Quality Package
Dev
Test (E2E, NFA)
trigger
test
Tests

18. Hello Testkube.
Your friendly cloud-native testing framework for Kubernetes
QAware | 18
■ Testkube natively integrates test orchestration and execution into Kubernetes and your CI/CD or
GitOps pipeline
■ Avoids vendor lock-in for test orchestration and execution in CI/CD pipelines
■ Makes it possible to decouple test execution from build processes; test engineers should be able to
run specific tests whenever needed
■ Makes it easy to run any kind of tests - functional, load/performance, security, compliance, etc. in
your clusters, without having to wrap them in docker-images or providing network access
■ Provides a modular architecture for adding new types of tests and executors
■ https://github.com/kubeshop/testkube

19. Demo Architecture and Testkube Concepts
QAware | 19
default
testkube
Testkube Dashboard
Webhook
Receiver
Testkube API Server
CRDs
CI/CD
System
Dev
Executors
Test
Test
Suite
Microservice
trigger
flux-system
run
Mongo
DB
NATS
Minio
S3
CLI
start
store
watch
Test
Trigger
SUT
Monitoring
System
Test
Source

20. lreimer/testkube-zap-demo
lreimer/hands-on-testkube

21. qaware.de
QAware GmbH
Aschauer Straße 32
81549 München
Tel. +49 89 232315-0
info@qaware.de
twitter.com/qaware
linkedin.com/company/qaware-gmbh
xing.com/companies/qawaregmbh
slideshare.net/qaware
github.com/qaware
Contact details ...