This document discusses rapid test automation techniques used at Spotify. It describes how manual testers can generate test cases and automation scripts quickly using tools like Graphwalker and Sikuli to get testing feedback within hours, enabling rapid and continuous testing. An example is provided where features of Spotify's HTML5 desktop client were tested automatically in this way. The document concludes by discussing opportunities for fully automated testing using tools like GUITAR and WebTestingExplorer.

1. Alexander Andelkovic, Spotify
alex@spotify.com STARWEST 2012

2. Introduction
Background
Rapid Test Automation
Example
Summary
Fully Automated Testing
Questions
1/21

3. Lives in Stockholm, Sweden
15+ years testing experience
Test lead at Spotify
Passionate about advanced test techniques
2/21

4. Our reality:
- Tougher business competition
- Rapid releases needed to stay competitive
- Shorter release cycles (Year > Month > Week > Day)
Problem:
Difficult to keep-up with test automation of new
functionality as stories accumulates after each sprint
Solution:
Use Rapid Test Automation for faster test feedback loops
3/21

5. Test automation(TA) done by test developer and manual tester
Activity Traditional TA Agile TA Rapid TA
Modeling Test developer Manual tester Manual tester
Test developer Test developer
Coding Test developer Test developer Manual tester
Test developer
Testing feedback Month(s) Week(s) Day(s)
4/21

6. Test automation(TA) done by test developer and manual tester
Activity Traditional TA Agile TA Rapid TA
Modeling Test developer Manual tester Manual tester
Test developer Test developer
Coding Test developer Test developer Manual tester
Test developer
Testing feedback Month(s) Week(s) Day(s)
4/21

7. Test automation(TA) done by test developer and manual tester
Activity Traditional TA Agile TA Rapid TA
Modeling Test developer Manual tester Manual tester
Test developer Test developer
Coding Test developer Test developer Manual tester
Test developer
Testing feedback Month(s) Week(s) Day(s)
4/21

8. Test automation(TA) done by test developer and manual tester
Activity Traditional TA Agile TA Rapid TA
Modeling Test developer Manual tester Manual tester
Test developer Test developer
Coding Test developer Test developer Manual tester
Test developer
Testing feedback Month(s) Week(s) Day(s)
4/21

9. Model-Based Testing used for auto generation of test procedures/vectors using models of system requirements
Agile TA Rapid TA
Finite State Machine(FSM) Extended Finite State Machine(EFSM)
y-Ed used to create models
Graphwalker (Model Based testing tool) used to traverse models 5/21

10. - Simple scripts(Python) developed in Sikuli test framework to
build test automation
- Manual tester writes own test automation code
- No need to make test automation stable only keep usable code
6/21

11. - Exploratory testing done while modeling and coding
- Stress testing can be done immediately or overnight
- Manual tester give early/instant feedback to test developer of test
cases suited for robust regression testing
7/21

12. Features of Spotify desktop client rewritten into HTML 5
End user shall have same experience of the new rewritten HTML 5
features(What’s New, Top List, Profile) in the Spotify desktop client
8/21

13. Browse What’s
New view
Main view
9/21

14. Play Top track
10/21

15. Search Track,
Artist, Album
11/21

16. Write Message
11/21

17. Send track and
message
11/21

18. 12/21

19. Init
Loop
End
13/21

20. 14/21

21. - Manual tester not dependant on test developer
- Test automation can be developed immediately when needed
- Valuable test feedback can be given within hour(s)
- Instant regression, performance and stress testing can be done
- Developers shows greather interest in testing
15/21

22. Fully Test Automation
the next TEST frontier?
16/21

23. - GUITAR is a model-based system for automated GUI testing
- Developed by Atif Memon, associate professor in the
Dept. of Computer Science at the University of Maryland
- Automated GUI-based test case generation
- Automated test case execution
- Automated visualization of test case execution
Atif Memon 17/21

24. - Automated event/screen ripping with crawling of an
entire application
- Overlaying of test cases onto screenshots(node and
edge drawing)
- Recording of manual test cases through Selenium
IDE and execution with GUITAR Replayer
- Support for Web, iPhone, Android, Java apps
18/21

25. - WebTestingExplorer is an Automated test case generator for
web front-ends
- Developed by Scott McMaster former student of Atif Memon
- Generates and replay test cases, looking for differences in
state
- Detects unexpected HTTP status codes and Javascript errors
during exploration
Scott McMaster
19/21

26. E-mail: alex@spotify.com
Graphwalker: http://graphwalker.org/
yED: http://www.yworks.com/en/products_yed_about.html
Sikuli: http://sikuli.org/
Web Guitar:
http://sourceforge.net/apps/mediawiki/guitar/index.php?title=WebGuitar
Web Testing Explorer:
20/21

27. 21/21