Experiences Of Test Automation
@ Spotify
Kristian Karl

December 2, 2013
Scaling Agile @ Spotify with Tribes, Squads, Chapters & Guilds, by Henrik Kniberg & Anders Ivarsson
Why automation?

•
•
•
•

We want to move faster
Quick dev feed-back
Continous Delivery
Continously monitor the
state of t...
What to automate?
Graphical user interface testing
Usability testing
Software performance testing
System testing
Functiona...
What to automate?

Graphical user interface testing
Our goals
•

Create automated end-user regression tests on 4 major
platforms
1) Desktop – Windows and OSX
2) iOS – iPhone ...
Our challenges
• Hard-to-test SUT
(Experiences of test automation: Case Study 1, An Agile Team’s Test
Automation Journey: ...
Model-based testing
• Models are the abstraction layer
• Testers design the automation using
models
• Developers implement...
App not
running
Exit app
Start app
Toggle
’Remember Me’

Close

Invalid credentials

Login view
displayed
Log out

Valid c...
public interface SimpleLogin {
public void e_Close();
public void e_Exit();
public void e_Init();
public void e_InvalidCre...
Demo
http://graphwalker.org/demo
Test automator

•
•
•
•

Professional Java developer.
Test experience is not mandatory.
Embedded in the squad (team).
Test...
Developers and developers

• Why not use developers for TA?
• Why use developers for TA?
• Test API’s
- Defined by TA
- Im...
18

Section name

Before
Test model

Test model

After
Test model

Test model

Test interface
+ wrappers

Test model

Test...
Before
String cmd = "android.view.View seekBarView =
solo.getView(com.spotify.mobile.android.ui.view.CancellableSeekBar.cl...
Supporting services

•
•
•
•
•

Continuous testing
QA Lab
TDS – Test Data Service
TRS – Test Result Service
Virtualization
Open source (and freeware) tools that we use

• yEd – editing FSM models [www.yworks.com]
• GraphWalker – generate test se...
Demo
Q&A
"Experiences Of Test Automation At Spotify" with Kristian Karl
"Experiences Of Test Automation At Spotify" with Kristian Karl
"Experiences Of Test Automation At Spotify" with Kristian Karl
"Experiences Of Test Automation At Spotify" with Kristian Karl
"Experiences Of Test Automation At Spotify" with Kristian Karl
Upcoming SlideShare
Loading in …5
×

"Experiences Of Test Automation At Spotify" with Kristian Karl

3,628 views

Published on

View webinar: http://www.eurostarconferences.com/community/member/webinar-archive/webinar-87-experiences-of-test-automation-at-spotify-

At Spotify, we want the manual testing effort to be focused as much as possible at feature testing, less on regression tests. But we still have to do regression. So, we tried to automate a big chunk of that. Regression tests are run on our Desktop, Android, iOS and WebPlayer clients, and also some backend services.

I will share with you how far we have come. What techniques, tools and methodologies we have tried. What experiences has been good, and what has been not that good.

Published in: Technology

"Experiences Of Test Automation At Spotify" with Kristian Karl

  1. 1. Experiences Of Test Automation @ Spotify Kristian Karl December 2, 2013
  2. 2. Scaling Agile @ Spotify with Tribes, Squads, Chapters & Guilds, by Henrik Kniberg & Anders Ivarsson
  3. 3. Why automation? • • • • We want to move faster Quick dev feed-back Continous Delivery Continously monitor the state of the product • Bug hunting From: ”Intelligent Test Automation” by Harry Robinson
  4. 4. What to automate? Graphical user interface testing Usability testing Software performance testing System testing Functional testing Load testing Volume testing Stress testing Security testing Scalability testing Sanity testing Unit testing Smoke testing Component testing API testing Regression testing Installation testing Maintenance testing Recovery and failover testing. Accessibility testing Monkey testing Integration testing
  5. 5. What to automate? Graphical user interface testing
  6. 6. Our goals • Create automated end-user regression tests on 4 major platforms 1) Desktop – Windows and OSX 2) iOS – iPhone and iPad 3) Android 4) Webplayer • • • Help testers, not replacing them Deliver automated regression tests for a feature as a part of definition of done Deliver short feedback loops to teams using Dashboards
  7. 7. Our challenges • Hard-to-test SUT (Experiences of test automation: Case Study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy Graham & Mark Fewster) • Maintenance of automation • Expectations • Flaky tests... or is it flaky SUT? • Testability • Test data, test environments • Supporting services
  8. 8. Model-based testing • Models are the abstraction layer • Testers design the automation using models • Developers implements the automation using the models as drivers
  9. 9. App not running Exit app Start app Toggle ’Remember Me’ Close Invalid credentials Login view displayed Log out Valid credentials Start app Main view displayed
  10. 10. public interface SimpleLogin { public void e_Close(); public void e_Exit(); public void e_Init(); public void e_InvalidCredentials(); public void e_Logout(); public void e_StartClient(); public void e_ToggleRememberMe(); public void e_ValidPremiumCredentials(); public void v_ClientNotRunning(); public void v_LoginPrompted(); public void v_WhatsNew(); }
  11. 11. Demo http://graphwalker.org/demo
  12. 12. Test automator • • • • Professional Java developer. Test experience is not mandatory. Embedded in the squad (team). Test automators form their own Guild
  13. 13. Developers and developers • Why not use developers for TA? • Why use developers for TA? • Test API’s - Defined by TA - Implemented by developers
  14. 14. 18 Section name Before Test model Test model After Test model Test model Test interface + wrappers Test model Test model QA/TA Java test interface JSON Strings ObjC test interface Client Devs Client
  15. 15. Before String cmd = "android.view.View seekBarView = solo.getView(com.spotify.mobile.android.ui.view.CancellableSeekBar.class, 0);"; cmd += "int[] xy = new int[2];"; cmd += "seekBarView.getLocationOnScreen(xy);"; cmd += "solo.clickOnScreen(xy[0] + 9 + (seekBarView.getWidth() - 18) * " + position + "f, xy[1] + seekBarView.getHeight() / 2.0f)"; BeanRemoteClient.sendToServer(cmd); After PlayerAuto player = Pages.remote(PlayerAuto.class); player.seekTrack(position);
  16. 16. Supporting services • • • • • Continuous testing QA Lab TDS – Test Data Service TRS – Test Result Service Virtualization
  17. 17. Open source (and freeware) tools that we use • yEd – editing FSM models [www.yworks.com] • GraphWalker – generate test sequences from models [graphwalker.org github.com/spotify/python-graphwalker] • Sikuli – image recognition tool [www.sikuli.org] • NuRemote - communicating with iOS client [github.com/nevyn/NuRemoting] • TestNG – group tests into suites and and run them [testng.org] • Java – code implementation • Jenkins – [test] job scheduler 24/7 [jenkins-ci.org]
  18. 18. Demo
  19. 19. Q&A

×