In today’s competitive world, more and more HTML5 applications are being developed for mobile and desktop platforms. Spotify has partnered with world-renowned organizations to create high quality apps to enrich the user experience. Testing a single application within a few months can be a challenge. But it's a totally different beast to test multiple world-class music discovery apps every week. Alexander Andelkovic shares insights into the challenges they face coordinating all aspects of app testing to meet their stringent testing requirements. Alexander describes an agile way to use the Kanban process to help out. He shares lessons learned including the need for management of acceptable levels of quality, support, smoke tests, and development guidelines. If you are thinking of starting agile app development or want to streamline your current app development process, Alexander’s experience gives you an excellent starting point.

1. How Spotify Tests World Class Apps
Alexander Andelkovic, Spotify
January 16, 2014

2. Agenda
Introduction
Background
Approval process
Future
Summary
Questions

3. Introduction – About me
- Lives in Stockholm, Sweden
- 15+ years testing experience
- Have a background in telecom and Med-Tech
- Test lead at Spotify
- Passionate about advanced test techniques

4. Background – Spotify Apps
- Apps(HTML5 & JS) integrated into Spotify music client(C++)
- Launched 30 Nov 2011 (at launch13 apps, today over 100
apps)
- Different categories of apps (discovery, concert, review,
social, lyrics)
- Manual QA process

5. Background – QA Challenges
- Spotify App = mini application
- Release high quality in no time
- Testing needs to scale over time(partners need to test)
- Stable test environment(new API, backend)
- Dependant on external development(partners)

6. Approval process - Introduction
- Steps that help partners ”release” high quality apps
- Approval steps: Concept > Development > Release > Update
- Average app development time 3 months
- Approval team around 10-15 persons

7. Approval process - Team
- Product owners(prioritize tasks, final calls)
- QA(manual testers)
- Test automators(regression test)
- Designers(concept, prototype)
- Release manager(silent/big bang release)

8. Approval process - Concept
- An app shall bring „valuable content‟ or something „unique‟ to
the Spotify experience.
- A good concept will save development time
- Summary, Mockups, Wireframes, Screenshots, Alpha version
- Designers gives valuable feedback
- No development should be started until concept is approved

9. Approval process - Development
- App development can start after concept approved and agreement
signed
- App should be submited for approval when ”production ready”
- QA gives thorough feedback on app quality
- An app is approved when ”all” issues are resolved

10. Approval process - Release
- When an app is approved a release date and
market(s) are decided
- PR involved(internal, external)
- Release manager inform partner when app is live
- Live app needs to be maintained

11. Approval process - Update
- Bugfixes, tweaks, small features
- Requires new App Submission
- App goes through the approval phase for Development again
- ”New” version treated as a new app concept(approval phase concept)

12. Approval process - Kanban
- “Just-in-time” approval process
- App submission enter pipeline
- Approval process(Concept > Development > Release > Update)
inside pipeline
- App submission approved/not approved emerges from the other
end of the pipeline
- Effective development process for identifying bottlenecks

13. Approval process - Support
- Stack overflow
- Internal tech support
- Keeps guidelines updated
- Best practises posted to developer blog

14. Approval process - Partners
- One man shows
- Web developers
- App developers
- Inhouse development

15. Approval process - Checklist
- Approximately 50 teststeps(manifest, navigation, performance,
security etc..)
- Should be run through by partner before app submission
- Session based testing used(Charter, timeboxed, debrief)
- Test session divided into smoke test and exploratory test(ET)
phase
- ET mindset crucial
- Pair testing performed on first submission to find as many bugs as
possible(reduces re-submissions)

16. Approval process – Test automation
- Regression vs bug finding test automation(TA)
- Robust TA a challenge(false positives)
- Empower partners to perform own TA
- App TA tests production environment

17. Approval process - Security
- App security tests performed by Security
Penetration Testers
- All ways of input needs to be tested
- Are HTML5 and javascript apps a security risk?
- Partner homepages

18. Approval process - Metrics
- Performance metrics(daily, weekly, monthly goals)
used to tweak approval process
- Enables greather understanding of user app
behaviour
- Drives A/B testing of appfinder
- Continous app content tweaking

19. Future
- Quantity vs quality
- More test automation(bug finding, crawler)
- More advanced apps(WebGL)
- App SDK‟s

20. Summary
- Developer performing tests early reduces app development time
- ”Tweak it” mindset crucial for an efficient approval process
- App develpment a new dicipline
- Apps are fun and easy try it out

21. Questions
alex@spotify.com

22. Thank you!
January 16, 2014