Jfokus 2023, Februar 2023, Stockholm, Schweden, Mario-Leander Reimer (@LeanderReimer, CTO @QAware). == Dokument bitte herunterladen, falls unscharf! Please download slides if blurred! == Continuous delivery is everywhere. Really?! Many teams still struggle to deliver well-tested product increments on a regular basis. Usually with the same old excuse: the (non)-functional tests are too complex and too expensive to implement thoroughly. But exactly the opposite is the case! In this talk, we briefly review the importance of early and regular testing of cloud-native applications and explain why monolithic CI pipelines are a dead end. We then show how easy it is to run integration, performance, security and acceptance tests continuously using Testkube directly on your Kubernetes cluster, fully integrated with a GitOps approach.

1. qaware.de
Quality in a Square.
K8s-native Quality Assurance with Testkube
Mario-Leander Reimer
mario-leander.reimer@qaware.de
@LeanderReimer
#Jfokus #CloudNativeNerd

2. QAware | 2
https://imgur.com/gallery/LGAZEqu
The Early
Code Monkey
The Monolith

3. QAware | 3
Monolithic systems were relatively
easy to test.
■ No distribution, no IPC
■ Homogene technology stack
■ Low infrastructure complexity
■ Managed infrastructure
■ Long release and test cycles
■ Developed by one team

4. QAware | 4
Microservice-based systems are complex.
Testing them is even more complex.
■ High distribution with various communication channels
■ Heterogeneous Technology Stacks
■ High infrastructure complexity with many components
■ New operating model with more responsibility for the developers
■ Short release cycles. Many teams.

5. 5
Mario-Leander Reimer
Managing Director | CTO
@LeanderReimer
#cloudnativenerd #qaware
#gernperDude

6. Perfect and Complete Continuous Delivery is not so easy!
QAware | 6
Continuous
Delivery
Low Risk
Releases
Less
Rework
Fast Time
to Market
Better
Products
Lower
Costs
Happier
Teams
Happier
Users
Loosely Coupled Architectures
Maintainable Code
Empowered Teams
Continuous Security from Day 1
Test
Automation
Continuous
Integration
GitOps
Deployment
Automation
Monitoring and Alerting

7. Holistic Software Product Quality is a lot of work!
QAware | 7
Software Product
Quality
(ISO 25010)
● Modularity
● Reusability
● Analysability
● Modifiability
● Testability
Maintainability
● Confidentiality
● Integrity
● Non-repudiation
● Authenticity
● Accountability
Security
● Adaptability
● Installability
● Replaceability
Portability
● Co-existence
● Interoperability
Compatibility
● Maturity
● Availability
● Fault Tolerance
● Recoverability
Reliability
● Time Behaviour
● Resource Utilization
● Capacity
Efficiency
● Completeness
● Correctness
● Appropriateness
Functional Suitability
● Operability
● Learnability
● UI Aesthetics
● Accessibility
Usability
Safety
Deployability
Operability

8. Monolithic, linear CI/CD pipelines are suboptimal and will
result in delayed feedback and long release cycles.
QAware | 8
Usually delayed until the end of sprint or the release.
Which one first? Functionality vs. Performance vs. Security?

10. A microservice architecture with many downstream
dependencies is complex and really hard to test.
QAware | 10
Cluster
Microservice
A
Microservice
B
Microservice
C
External
System X
External
System Y
Team A Team C Team B Unknown

11. Why not run (non)-functional tests against a
cloud-native microservice architecture
continuously, or triggered on the cluster itself?

12. Product Assembly Line

13. Core Principles and Concepts of a Product Assembly Line
QAware | 13
■ Separation of Concerns
– Do one thing, and do one thing well.
– Separate and explicitly model distinct development lifecycle phases: Build, CI, Deploy, E2E Test, …
■ Choreography over Orchestration
– Codified or configured downstream dependencies in the pipelines and triggers must be avoided.
– Loose coupling. Tests and Suites do not need knowledge about components or environments
– The product assembly line (PAL) is event-driven: time-based, commit based, deployment-based,
release-based, on-demand
■ GitOps Principles
– All individual and E2E product tests as well as required components are GitOps managed. Auditable.
Versioned. Easy Time Travel (e.g. revert to previous version)
– Current Test BOM (Bill of Material) can be determined via individual state repositories
■ Holistic Quality
– We need a Single source of (quality) truth!
– All relevant quality attributes are continuously aggregated and validated.

14. Conceptual PAL Architecture
QAware | 14
Packages
Package
publish
update
Run
deploy
watch
Deploy
watch
Dev GitOps
Build
push
Checkout Build Test Quality Package
Dev
Test (E2E, NFA)
trigger
test
Tests

15. Hello Testkube.
Your friendly cloud-native testing framework for Kubernetes
QAware | 15
■ Testkube natively integrates test orchestration and execution into Kubernetes and your CI/CD or
GitOps pipeline
■ Avoids vendor lock-in for test orchestration and execution in CI/CD pipelines
■ Makes it possible to decouple test execution from build processes; test engineers should be able to
run specific tests whenever needed
■ Makes it easy to run any kind of tests - functional, load/performance, security, compliance, etc. in
your clusters, without having to wrap them in docker-images or providing network access
■ Provides a modular architecture for adding new types of tests and executors
■ https://github.com/kubeshop/testkube

16. Demo Architecture and Testkube Concepts
QAware | 16
default
testkube
Deployment
Testkube Dashboard
Webhook
Receiver
Testkube API Server
CRDs
CI/CD
System
Dev
Executors
Test
Test
Suite
Deployment
trigger
flux-system
run
Mongo
DB
NATS
Minio
S3
CLI
start
store
watch
Test
Trigger
SUT
Monitoring
System
Test
Source

17. kubeshop/testkube
lreimer/hands-on-testkube

18. Architectural fitness functions continuously validate the
(non)-functional requirements aka -illities of a system.
QAware | 18
https://www.thoughtworks.com/de/radar/techniques/architectural-fitness-function

19. Some Architectural Fitness Function Examples
QAware | 19
https://www.thoughtworks.com/de/insights/articles/fitness-function-driven-development
describe "Performance" do
it "completes a transaction in under 2 seconds" do
expect(k6.check_transaction_round_trip_time()).to < 2
end
it "has less than 10% error rate for 10000 transactions" do
expect(k6.check_error_rate_for_transactions(10000)).to < .1
end
end
describe "Security" do
describe “Static Analysis” do
it "should not have plaintext secrets in codebase" do
expect(code.has_secrets_in_codebase()).to_not be(true)
end
end
describe “Dynamic Analysis” do
it "should not have any of the OWASP Top 10" do
expect(zap.has_owasp_top_10_vulnerabilities()).to be(false)
end
end
end

20. Conceptual PAL Architecture
QAware | 20
Packages
Package
publish
update
Run
deploy
watch
Deploy
watch
Dev GitOps
Build
push
Checkout Build Test Quality Package
Dev
Test (E2E, NFA)
trigger
test
Tests
Quality
Cockpit
report
PO
report
Build Status
Code Quality
Test Results
fitness
test
promote ?

21. qaware.de
QAware GmbH
Aschauer Straße 32
81549 München
Tel. +49 89 232315-0
info@qaware.de
twitter.com/qaware
linkedin.com/company/qaware-gmbh
xing.com/companies/qawaregmbh
slideshare.net/qaware
github.com/qaware
Contact details ...