The document summarizes experiences with test automation and discusses three test automation frameworks developed over 10 years for marketplace systems. It describes the process of contributing chapters to a book on test automation experiences. The frameworks aimed to automate an increasing number of tests to support agile development practices. The first framework developed tests quickly under time pressure with a thin abstraction layer. The second framework focused more on the test tool than tests. The third framework aimed for a thicker abstraction layer but architects became bottlenecks. ROI analysis showed savings from automation increased with larger test batches.

1. Testautomatiseringserfarenheter

3. Presentation
Michael Albrecht Lars Wahlberg (Aguil)

4. Vad innehåller boken?

5. Våra bidrag
• Michael Albrecht
– Anecdote 29.12 Cooperation Can Overcome
Resource Limitations
• Lars Wahlberg
– Chapter 18 Automated Tests for Marketplace
Systems: Ten years and Three Frameworks
Nu blir det vissa slides på Engelska !!!

6. Framtagning av en bok
• Hur gick det till
– Hur vi blev kontaktade
• Arbetet med att ta fram bok
– Massor med reviewer & uppdateringar
– Tekniska saker som att pillra med bilder
• Legal process
– Skriva på kontrakt osv
• Känslan att få boken (tillslut)

7. Första stegen

8. Anecdote 29.12 Cooperation Can
Overcome Resource Limitations

9. Before the agile hype
• Sit together
• Daily status meetings
• Learn some basic programming (same
language as the developers)
• Select tools already used by the developers
for unit testing if possible
• Create semi-automatic and automatic tests
• Learn from the past
• Get going!

10. Technical automater
• 55% a good tester
• 35% development skills
• 10% business or domain knowledge

11. .and the rest of the story?
• http://www.addq.se/nyheter/back-to-the-
future-an-agile-story-from-the-past/
• http://www.automatedtestinginstitute.com

12. Chapter 18 Automated Tests for Marketplace
Systems: Ten years and Three Frameworks
• Market Place Systems
• Automated Tests
• Three frameworks
• ROI

13. Market Place Systems (MPS)
Standby
Trading Clients Primary
Protocol X
Gwy A Query / History
Server(s)
Trading Clients
Gwy B
Protocol Y
Matching
Gwy C Engine
Market Data
Gwy D Central
Data
Admin Client
Typical instruments: Equities, Commodities, Bonds, Derivatives …

14. Drivers for large nr of tests …
• Complex functionality
• Interference of functionality
• Critical functionality
– Wrong auto match L
– Wrong transparency L
– Possible to “game” the system L
• High uptime requirements
• Many customers (adaptations of a product)

15. Why automate?
• Large nr of tests
• Development methods – many releases!
– Incremental (10 years ago)
– Agile (now days)
• Time to market important (rapid deliveries)
– Enhancements (new business opportunity)
– Patch (maintenance)
Solution => automate tests ...

16. Development of Test Framework
Framework for automated tests
Abstraction Layer
System
Test
Scripts
Test Configuration
Engine
(ex JUnit)

17. Abstraction Layer – Example
Changes in the protocol can be handled in
Test Script
the abstraction layer, instead of rewriting
Enter Buy Order (25 SEK) ~100 of tests scripts …
Enter Sell Order (24 SEK)
Verify Trade (25 SEK) Enter Order (price, isBuy, …)
• Create Enter Order Message
• Fill in Price X
Enter Buy Order(price)
• Fill in Buy or Sell
Enter Order (price, True, ...)
• Fill in …
• Send Transaction

18. Abstraction of Configuration
• Predicates can be used
Test Script If not found =>
myList.add(new PredicateInstrument(“Equity”) )
Test Skipped !
myList.add(new PredicateCurrency(“USD”) )
Not same as Failed !
myList.add(new PredicateTrader(myTrader1) )
myList.add(new PredicateTrader(myTrader2) )
OrderBook myOb = TestConfig.getAnyOb(myList)
Hint: Use TDD to design the Abstraction Layer !

19. Development of Test Tool
• Abstraction Layer is important
– Thin / Thick? Requirements on people skills!
– Who can modify? Who will maintain?
– When tests increased problems will be revealed!
• Abstraction against test data
– Hard coded in a file, version controlled with tests
– Predicates may be used to increase robustness
Tip: Very good reference go to => www.stickyminds.com
Search for “mexus” (author Lars-Ivar Sellberg)

20. Development of Tests
Automatic Tests
5000
4500 Actual
Planned
4000
3500
3000
Tests
2500
2000
1500
1000
500
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Week

21. Test Execution (then)

22. Test Execution (then)
Now it works again! :)
Now it works ! :)
TestCase written Oops, it fails! :
before implementation ( Regression tests are
good !
Oops, vacation!

23. Framework - 1
• Developed under much time pressure 1998
• Tests written in Java
– New language, attracted developers to test
– Thin Abstraction Layer
• Own Test Engine (JUnit not available)
• Predicated used
• Simulations used (later more on that)
• After ~1 year: 2000 tests, ~3 year: 15 000 tests
• A real success !!!

24. Framework - 2
• Developed 1999 based on success of -1
• Development took some time …too much
– Another system => new abstraction layer, still thin
– Predicates not used
– Switched to JUnit
– Focus on Tool instead of tests … few tests L
• Heavy investment during 2002
– Partly due to one senior Manager J
– Larger team => 50 000 tests after 3 years!

25. Framework - 3
• Developed around 2003
• Based on ideas from mostly from 1
– But should be better! Func & Non Func tests, Preds
– Thick Layer – Tests in XML action language
• Test Tool Architects became bottlenecks
– Difficult to do modifications of abstraction layer L
• Re-done, Thin layer & No Predicates
• Now ~ 20 000 Tests

26. Effort dev manual test 2 h/test
Effort dev automatic test 2 h/test
ROI - Savings
Test Tool Maintanence 3000 h/year
Effort Execute manual test 0,1 h / test
Effort Execute automatic test 8 h/batch
Saving of Automization
150
100
50
0
savings in %
Yearly
1100
1300
1500
1700
1900
500
700
900
-50
Monthly
-100
Weekly
-150
Daily
-200
-250
-300
-350
nr of tests in batch

27. Utlottning av böcker !