A look at testing as it used to be and the way it can be done in an enterprise where the benefits of continuous integration combined with multi-level test automation are understood.

1. 1
Towo Toivola
Lead Quality Engineer, F-Secure Corporation

4. Bonus:
RicardoKekki
© F-Secure Confidential4
SeniorQualityEngineer

5. Are you in?
5

6. ROI!
6

7. ”Kyseenalaistakaakaikki
mitäsanon,mutta
kyseenalaistakaa
avoimella,älkää
jääräpäisellämielellä”
7

8. Contents
• In the olden days..
• What testing used to be
• What testing used to be
for
• Test Automation
compared to manual
testing
• New world of working
together
8
• New value of testing:
CI+TA=status
• Great advantages of TA
• Modern Test
Automation is..
• Imagine 100 people
working on a Trunk..
• ’Done includes TA’

9. In the Olden
Days..
9

10. Intheoldendays..
Testing was..
 Manual
 Boring
 Unrespected
 Tedious
 Slow
 Expensive
 Out-sourced
 Off-shored
10

11. Intheoldendays..
Organizations used testing for..
1. Find out what work is still needed
2. Verify that we are good to release
Practically this meant:
 Find list of bugs we may decide to fix later
 Last mile before releasing, a ’check for confidence’
- Usually a preventer of release..
11

12. Intheoldendays..
Feedback cycles were weeks or months
More calendar time means:
 More effort in between
 More uncertainty
 More investigation cost
 More discord between testing and programming
 More open bugs
 More wasted effort in testing and programming
 More requirements change
 Less customer feedback
12
How is this
supposed to
work?
Reproduce it
with the latest
version.
We need PSG
decision on
bugfixing.
EXPENSIVE!

13. Test
Automation
13

14. ManisSuperior to
theMachine
 Works around problems
 Common sense
 Creativity
 Understanding of the user
 Understanding of priorities
 Understands change
 Looking outside the box
 The most important bugs are found in the requirements and
assumptions
Towo Toivola 2009
This was rarely
leveraged in the
olden days..

15. TheMachineis
Superior toMan
 Automation is faster
 Automation has more attention to detail
 Automation can run through more data
 Automation can execute more combinations
 Automation can measure more exactly
 Automation does not get tired
 Automation works through the night
 Automation provides exact logs
Software
Tool
Test
Towo Toivola 2009

16. The New Age of
Cooperation
and Results
16

17. NewAge..
TheValueofTesting
CI + TA
= status
17

18. © F-Secure Confidential18
Test on
multiple
levels
”Are we
stable? Can we
move on?”
Does it make
sense to perform
exploratory
testing now?
Can we give
this to others
now?
NewAge..
CI+TA=status

19. Legend
Scanbox version maturity
state
Environment / tool
Activity/ test
Decision
Scanbox build smoketests Jenkins + Xingu
”Not broken”
Flow Time
<10 min
”Stable”
Binary Repository
Scanbox regression TA Jenkins + Xingu <120 min
Malware detection rate TA Jenkins + Orange Scanbox performance &
reliability TA
Jenkins + Xingu
”Super Stable”
Promote to Alfa
”Alfa”
Nightly / on-
demand
Deployment, Long test,
monitoring, perf check
Alfa environment in AWS
User group
Other dev teams
1-3 times per
week
1-10 days
Promote to Beta
”Beta”
Deployment, monitoring
Beta environment in AWS
External beta use
1-10 days
Promote to Production
Release
Out.
From the test
automation
radiator we see
how our release
process is
doing.
Feedback
cycles are
minutes!

20. GreatAdvantagesof
TestAutomation
 ”If it doesn’t have test automation, I will not touch it.”
 ”How can I fix it if I can’t tell if it works?”
 Brings the skills of software engineers and test engineers
together
 Shifts focus from ’How’ to ’Why’
 Guides architecture and scope decisions
20

21. BetterthanSpecs!
 It is specific
 It is unambiguous
 It is version controlled
 You can verify if software meets it
 You can verify your ’fix’ is an improvement
 It is written by people who know the software
 You can go and fix it if it is broken
21

22. ModernTest
Automation
 Fast
 Disciplined
 Jointly owned
 Business critical
 The priority!
22
Few things are
as expensive as
broken test
automation.

23. Howdoyougetit?
23

24. Are some of
your projects
large?
24

25. Facebook:
© F-Secure Confidential25

26. Facebook:
© F-Secure Confidential26

27. Imagine 100
people working
on a trunk..
27

28. 28
My local code
My system
tests
Team system
tests
I am mister
programmer
My unit tests
Team code
Project code
Project system
tests
Good version
Exploratory
system tests
Beta version
GIT
Jenkins
Eucalyptus
Python
Jira

29. 29

30. Thankyoufor
yourtime.
PLEASE FEEL ENCOURAGED TO APPROACH ME FOR A DISCUSSION.
May 15, 201430