This document outlines 10 secret weapons for performance testing in an agile environment. It discusses making performance explicit, having performance testers be part of the development team, using customer requirements to drive performance tests, taking a disciplined approach to analyzing test results, starting performance testing early in projects, iterating rather than just incrementing, automating tests, treating performance test code like application code with unit tests, implementing continuous performance testing, and getting frequent feedback on performance.

1. Top ten secret weapons for performance testing in an agile environment patrick.kua@thoughtworks.com (and alistair.jones@thoughtworks.com) © ThoughtWorks 2011

2. Make Performance Explicit © ThoughtWorks 2011 1

3. So that I can make better investment decisions As an investor I want to see the value of my portfolio presented on a single web page must have “good” performance, less than 0.2s page load for about 10,000 concurrent users © ThoughtWorks 2011

4. © ThoughtWorks 2011 So that investors have a high-quality experience as the business grows As the Operations Manager I want the portfolio value page to render within 0.2s when 10,000 users are logged in

5. One Team © ThoughtWorks 2011 2

6. Team Dynamics © ThoughtWorks 2011

7. Performance Testers Part of Team © ThoughtWorks 2011

8. © ThoughtWorks 2011

9. Performance Testers Part of Team © ThoughtWorks 2011

10. Pair on Performance Test Stories © ThoughtWorks 2011

11. Rotate Pairs © ThoughtWorks 2011

12. Customer Driven © ThoughtWorks 2011 3

13. What was a good source of requirements? © ThoughtWorks 2011

14. © ThoughtWorks 2011 Existing Pain Points

15. An example... © ThoughtWorks 2011

16. So that we can budget for future hardware needs as we grow As the data centre manager I want to know how much traffic we can handle now © ThoughtWorks 2011

17. Another example © ThoughtWorks 2011

18. © ThoughtWorks 2011 So that we have confidence in meeting our SLA As the Operations Manager I want to ensure that a sustained peak load does not take out our service

19. Personas © ThoughtWorks 2011

20. Who is the customer? © ThoughtWorks 2011 Investors Marketing End Users Power Users Operations

21. Discipline © ThoughtWorks 2011 4

22. © ThoughtWorks 2011 Observe test results What do you see? Formulate an hypothesis Why is it doing that? Design an experiment How can I prove that’s what’s happening? Run the experiment Take the time to gather the evidence. Is the hypothesis valid? Change the application code Safe in the knowledge that I’m making it faster

23. ?????????? © ThoughtWorks 2011

24. © ThoughtWorks 2011 Observe test results Saw tooth pattern (1 minute intervals) Formulate an hypothesis Directory structure of (yyyy/mm/minuteofday)?. Slow down due to # of files in directory? Design an experiment 1 directory should result in even worse performance... Run the experiment We ran the test… Is the hypothesis valid? Change the application code

25. One Directory © ThoughtWorks 2011

26. Play Performance Early © ThoughtWorks 2011 5

27. © ThoughtWorks 2011 End Start Other projects start performance testing here End Start Agile projects start performance testing as early as possible

28. Iterate Don’t (Just) Increment © ThoughtWorks 2011 6

29. © ThoughtWorks 2011

30. We Sashimi © ThoughtWorks 2011

31. Sashimi Slice By... Presentation © ThoughtWorks 2011

32. © ThoughtWorks 2011 So that I can better see trends in performance As the Operations Manager I want a graph of requests per second

33. © ThoughtWorks 2011 So that I can better see trends in performance As the Operations Manager I want a graph of average latency per second

34. © ThoughtWorks 2011 So that I can easily scan results at a single glance As the Operations Manager I want a one page showing all results

35. Sashimi Slice By... Scenario © ThoughtWorks 2011

36. © ThoughtWorks 2011 So that we never have a day like “October 10” As the Operations Manager I want to ensure that a sustained peak load does not take out our service

37. © ThoughtWorks 2011 So that we never have a day like “November 12” As the Operations Manager I want to ensure that an escalating load up to xxx requests/second does not take out our service

38. Automate, Automate, Automate © ThoughtWorks 2011 7

39. © ThoughtWorks 2011 Automated Compilation Automated Tests Automated Packaging Automated Deployment

40. Automation => Reproducible and Consistent Automation => Faster Feedback Automation => Higher Productivity Why Automation? © ThoughtWorks 2011

41. © ThoughtWorks 2011 Automated Test Orchestration Automated Analysis Automated Scheduling Automated Load Generation Automated Application Deployment Automated Result Archiving

42. Continuous Performance Testing © ThoughtWorks 2011 8

43. © ThoughtWorks 2011

44. Application Build Pipelines © ThoughtWorks 2011 Performance

45. © ThoughtWorks 2011

46. Test Drive Your Performance Test Code © ThoughtWorks 2011 9

47. V Model Testing © ThoughtWorks 2011 Slower + Longer Performance Testing Speed Fast http://en.wikipedia.org/wiki/V-Model_(software_development)

48. We make mistakes © ThoughtWorks 2011

49. V Model Testing © ThoughtWorks 2011 Slower + Longer Performance Testing Speed Unit test performance code to fail faster Fast http://en.wikipedia.org/wiki/V-Model_(software_development)

50. Fail Fast! © ThoughtWorks 2011 Fast feedback! Faster learning Faster results

51. Classic Performance Areas to Test © ThoughtWorks 2011 Analysis Information Collection Presentation Publishing Visualisation

52. Get Feedback © ThoughtWorks 2011 10

53. Frequently (Weekly) Showcase © ThoughtWorks 2011 Here is what we learned this week....

54. Frequently (Weekly) Showcase © ThoughtWorks 2011 And based on this... We changed our directory structure.

55. Frequently (Weekly) Showcase © ThoughtWorks 2011 Should we do something different knowing this new information?

56. List of All Secret Weapons Make Performance Explicit One Team Customer Driven Discipline Play Performance Early Iterate Don't (Just) Increment Automate, Automate, Automate Test Drive Your Performance Code Continuous Performance Testing Get Feedback © ThoughtWorks 2011

57. Photo Credits (Creative Commons licence) Barbed wire picture: http://www.flickr.com/photos/lapideo/446201948/ Eternal clock: http://www.flickr.com/photos/robbie73/3387189144/ Sashimi from http://www.flickr.com/photos/mac-ash/3719114621/ Questions © ThoughtWorks 2011