Prezentacja z VI edycji Quality Excites. Tomasz i Dawid przedstawili ewolucję pracy testera, która w dzisiejszych czasach nierzadko wymaga umiejętności tworzenia i utrzymania odpowiednich środowisk do testów, do których stworzenia potrzebna jest wiedza na poziomie DevOpsa. Celem wystąpienia było przestawienie przykładowego szkieletu takiego środowiska do testów oraz narzędzi, które powinny się znaleźć w przyborniku TestOpsa.

1. SQA in a
TestOps era

2. SQA in a
TestOps era
Tomasz
Janiszewski
Dawid
Pacia

3. THE TRADITIONAL
MODEL OF
SOFTWARE
TESTING

4. THE TRADITIONAL MODEL OF SOFTWARE
TESTING
TESTS
QUALITY ASSESSMENT
RUN TESTS
SYSTEM UNDER TEST
ANALYSIS
PASS? FAIL?

5. “To improve is to change; to be
perfect is to change often.”
Winston Churchill

6. ➜ Facebook users send on average over
31 million messages and view 2.77
million videos every minute
➜ Twitter users generates 347,222
Tweets each minute – or 21 million
Tweets per hour
➜ Google provides search results for
over 1 billion search queries each day
➜ Amazon deploys new software to
production every 11.6 seconds.
The times has changed...

7. Let’s start with the first
set of slides
1.
TRANSITION
HEADLINE

8. EXPOSURE
CONTROL
TEST CASES AND
MONITORS
THE TESTOPS MODEL OF SOFTWARE
TESTING
EXPOSED USERS
AND SYSTEMS
SYSTEM UNDER TEST
PRODUCTION USERS
AND SYSTEMS
ACTIVE
MONITORS
ANALYSIS
QUALITY ASSESSMENT
PRODUCTION
USAGE
TELEMETRY DATA
KPIs / PATTERNS

9. CONTINUOUS
BUILD
CODE CHANGE /
FIX
SCM
(Git, Svn,
Bazaar ;)
NEW BUILD
(Ant, Maven,
Gradle, etc.)
CI TOOLS
(Jenkins,
Bamboo,
Travis CI,
Ansible,
etc.)
CONTINUOUS
VALIDATION
TEST
ENVIRONMENTS
BU
ILD
D
EPLO
YM
EN
T
TEST RESULTS
CONTINUOUS
DELIVERY
PRODUCTION
ENVIRONMENTS
CONTINUOUS
MONITORING
MONITORING
(Kibana, Grafana,
Scalyr, Cacti,
Amazon
CloudWatch, etc.)
SYNTHETIC /
SEMANTIC
MONITORING

10. ➜ Synthetic monitoring (or semantic
monitoring) runs a subset of an
application's automated tests against
the live production system on a regular
basis.
Continuous Monitoring

11. ➜ Integration with DevOps
infrastructure
➜ What tests to run and when
➜ No need to stand up the whole
system to test
TestOps: What Does This
Mean?

12. WHY SHOULD
WE CARE SO
MUCH?

13. FAKAPY
● Zgłaszanie błędu w komentarzy w Jirze - 500 klientom nie działało menu
nawigacyjne po aplikacji
● Dobrze zorganizowane glow w Gicie - pull req, praca miedzy zespolami
● Szkiletet
● Jak pisac kod (python)
● Nazwy obiektow!
● Re-uzywalnosc kodu
● Korzystanie z paczek
● Pop
● Pliki readme
● Jak unikac zlych praktyk
● Konwencja komitowanie (a nie ‘fixed small issue’)
● Konsekwencja pracy przez 2-mc na wlasnym branchu
● Dlaczego nie tylko produkt powinien byc testowany, ale również same
testy

14. BUT IN A WORLD OF FALSE...
False
Negatives
False
Positives
The test is marked as Failed,
when the functionality is
working
The test is marked as Passed,
when the functionality is not
working.

15. ...YOU CAN ENCOUNTER SOMETHING WORSE
False
Negatives
False
Positives
Flaky
Tests
TESTS FAILING
INTERMITTENTLY

16. RISKVALUE
CONTINUOUS: VALUE vs RISK
PLAN CODE BUILD TEST RELEASE DEPLOY OPERATE
COLLABORATION
AGILE DEVELOPMENT
CONTINUOUS INTEGRATION
CONTINUOUS DELIVERY
DEVOPS

17. Dawid Pacia
“More automatically,
more continuously,
more risky...”

18. TESTOPS
TOOLBOX
AND BEST
PRACTICES

19. GIT PROPER MODEL
19
time / commits
master
development
feature 1
feature 2
release
The Great
QA Wall

21. Pipelines
pipeline {
agent none
stages {
stage('error') {
steps {
parallel(
"Wait": {
sleep 1
},
"NodeExec": {
node(label: 'master') {
isUnix()
}
…
}
stage('ExternalJob') {
steps {
build(job: 'TestMulti1', propagate: true)
}
...
Example

22. But here it comes… Blue Ocean!

23. ➜ You can use compose to bring up
any number of instances of the
services in your stack, example:
> docker-compose up --scale
web=10 database=3
This is great for testing database
clusters or load balancers, or
resiliency with Pumba
Docker - Scalability Testing

24. ➜ Docker can be useful for testing different
system configurations like for example different
DB backends:
○ Bring up new stack (with MySQL backend),
Seed with data, Run API tests, Tear down
stack.
○ Bring up new stack (with Postgres
backend), Seed with data, Run API tests,
Tear down stack.
○ Report on any differences in test results
between MySQL and Postgres backends.
○ Integrates easily with Jenkins !
Docker - Configuration
Testing

25. Selenium Grid
RIP

26. Regression
Tests
Jenkins CI
Real-time reporting of Selenium tests on Kibana
UI
Tests
Acceptance
Tests
Apache Kafka
Elasticsearch Logstash
Jobs
Logs
Logs
Transformed
documents
Kibana
Queryresults

27. Final Results

28. ➜ Setup overview
○ Configure Apache JMeter (through user.properties file) to
Create a CSV File:
/opt/apache/jmeter/apache-jmeter-3.2/bin/jmeter -n -t Basic Elasticsearch
Test.jmx -l ./results.csv
○ Configure and Run Logstash
○ Visualize the Results
Real-time reporting of JMeter results on Kibana

29. Final Results

30. ➜ TestOps provides a unique aspect for
automated testing
➜ It works with the Dev team to set up
CI/CD/CM
➜ It creates a cloud or virtual technology to set
up a testing environment on demand
➜ It plays a vital role to integrate the automated
testing with Dev team for CI
➜ Explore and investigate!
Takeaway

31. Thanks!
Any questions?
You can find us at:
janiszewski.tomasz@aeon.network
pacia.dawid@aeon.network
web:
aeon.network