Test Automation presentation for Auckland Software Craftmanship group

1. 7 Years of Test
Automation
!
@mikeb2701

2. Assumptions
• Testing is important
• Automating testing is important

3. Types of tests
• Static Analysis
• Unit Testing
• Integration Testing
• Acceptance Testing!
• Performance Testing
• Testing in Live
• Exploratory Testing

5. The acceptance test suite as a
whole both verifies that the
application delivers the business
value and guards against
regressions or defects that break
preexisting functions of the
application.
— Humble, Farley, Continuous Delivery, 2010, Chapter 8.

6. An acceptance test may only
drive and assert behaviour in
the system through interfaces
available to the system’s users.

7. Writing Acceptance
Tests

8. @RunWith(AcceptanceTestRunner.class)
@Multithreaded
@AcceptanceTest(id = 29, flavour = {Flavour.PLACE_ORDER, Flavour.API})
public class PlaceOrderAcceptanceTest extends DslTestCase {
!
@Before
public void beforeEveryTest() {
!
adminAPI.createInstrument("name: instrument”,
"orderQuantityIncrement: 0.1”,…);
registrationAPI.createUser("user", "balance: 100000");
mtfMemberAPI.createMtfMemberAndFixSession("marketMaker");
publicAPI.login("user");
mtfFixAPI.login(“marketMaker");
} // end
!
@Test
public void shouldRejectOrderIfNotLoggedIn() {
publicAPI.logout();
publicAPI.placeOrder(“instrument",
"side: buy", "quantity: 1.2”,
"timeInForce: ImmediateOrCancel”,
"unauthorised: true");
!

9. @RunWith(AcceptanceTestRunner.class)
@Multithreaded
@AcceptanceTest(id = 29, flavour = {Flavour.PLACE_ORDER, Flavour.API})
public class PlaceOrderAcceptanceTest extends DslTestCase {
!
@Before
public void beforeEveryTest() {
!
adminAPI.createInstrument("name: instrument”,
"orderQuantityIncrement: 0.1”,…);
registrationAPI.createUser("user", "balance: 100000");
mtfMemberAPI.createMtfMemberAndFixSession("marketMaker");
publicAPI.login("user");
mtfFixAPI.login(“marketMaker");
} // end
!
@Test
public void shouldRejectOrderIfNotLoggedIn() {
publicAPI.logout();
publicAPI.placeOrder(“instrument",
"side: buy", "quantity: 1.2”,
"timeInForce: ImmediateOrCancel”,
"unauthorised: true");
!

10. @RunWith(AcceptanceTestRunner.class)
@Multithreaded
@AcceptanceTest(id = 29, flavour = {Flavour.PLACE_ORDER, Flavour.API})
public class PlaceOrderAcceptanceTest extends DslTestCase {
!
@Before
public void beforeEveryTest() {
!
adminAPI.createInstrument("name: instrument”,
"orderQuantityIncrement: 0.1”,…);
registrationAPI.createUser("user", "balance: 100000");
mtfMemberAPI.createMtfMemberAndFixSession("marketMaker");
publicAPI.login("user");
mtfFixAPI.login(“marketMaker");
} // end
!
@Test
public void shouldRejectOrderIfNotLoggedIn() {
publicAPI.logout();
publicAPI.placeOrder(“instrument",
"side: buy", "quantity: 1.2”,
"timeInForce: ImmediateOrCancel”,
"unauthorised: true");
!

11. @RunWith(AcceptanceTestRunner.class)
@Multithreaded
@AcceptanceTest(id = 29, flavour = {Flavour.PLACE_ORDER, Flavour.API})
public class PlaceOrderAcceptanceTest extends DslTestCase {
!
@Before
public void beforeEveryTest() {
!
adminAPI.createInstrument("name: instrument”,
"orderQuantityIncrement: 0.1”,…);
registrationAPI.createUser("user", "balance: 100000");
mtfMemberAPI.createMtfMemberAndFixSession("marketMaker");
publicAPI.login("user");
mtfFixAPI.login(“marketMaker");
} // end
!
@Test
public void shouldRejectOrderIfNotLoggedIn() {
publicAPI.logout();
publicAPI.placeOrder(“instrument",
"side: buy", "quantity: 1.2”,
"timeInForce: ImmediateOrCancel”,
"unauthorised: true");
!

12. @RunWith(AcceptanceTestRunner.class)
@Multithreaded
@AcceptanceTest(id = 29, flavour = {Flavour.PLACE_ORDER, Flavour.API})
public class PlaceOrderAcceptanceTest extends DslTestCase {
!
@Before
public void beforeEveryTest() {
!
adminAPI.createInstrument("name: instrument”,
"orderQuantityIncrement: 0.1”,…);
registrationAPI.createUser("user", "balance: 100000");
mtfMemberAPI.createMtfMemberAndFixSession("marketMaker");
publicAPI.login("user");
mtfFixAPI.login(“marketMaker");
} // end
!
@Test
public void shouldRejectOrderIfNotLoggedIn() {
publicAPI.logout();
publicAPI.placeOrder(“instrument",
"side: buy", "quantity: 1.2”,
"timeInForce: ImmediateOrCancel”,
"unauthorised: true");
!

13. DSL
TradingUI
PublicAPI
AdminAPI
Test
MtfMemberAPI
LoginPanel
Driver
Selenium
BalancesPanel
Driver
InstrumentPanel
Driver
JavaAPI
Driver
AdminAPI
Driver
MtfFixDriver
XML/HTTP
QuickFIX
Drivers

14. Time Travel

15. Service
Service
Service
Service
Service
Time
Machine
Test
Time Travel To
Time Delta
Get Current
Time

16. @RunWith(AcceptanceTestRunner.class)
@AcceptanceTest(id = 5194, flavour = {Flavour.PLACE_ORDER, Flavour.API})
@Multithreaded(tardis = true)
public class CancelAndReplaceAcceptanceTest extends DslTestCase
{
@Before
public void setUp()
{
dsl.forceToUtcTimeZone();
dsl.enableTimeMachine();
!
dsl.createTimePoint("name: origin”,
"value: <next weekday>");
dsl.createTimePoint("name: marketOpen",
"value: origin plus 1 weekdayCalendarOpenOffset”);
!
dsl.waitUntil("marketOpen");

17. Pro-Tip:
Isolate as much as
possible

18. Simple DSL
https://github.com/LMAX-Exchange/Simple-DSL

19. Acceptance Testing
(Running)

20. Parallel RTA
Sequential
Sequential
Sequential
RTA
RTA
RTA
Scotty
Romero
Server
Jenkins

23. Intermittency
avp.wikia.com

24. Sources of Intermittency
• Bad code (under test)
• Bad test
• Environmental

25. Just Fix It!

31. Acceptance Test DB

34. Wrap-Up
•Write them well
•Run them often
•Keep them green

35. Thank you
Q&A