Agile teams move faster when cycle times are short and code deployments are frequent. To release often, a robust suite of automated tests is a must-have. Tests are the safety net that enables fearless change. Throughout a software system's lifespan, its test suite grows, evolves, and decays. Left unchecked, test execution times increase and non-deterministic failures erode confidence. Ultimately, the test suite that once served as a change-enabler becomes an anchor, grinding progress to a halt. Scaling a test suite is complex and difficult—and vital to successful organizations. Drawing from experience in the trenches, Ryan Scott describes real-world examples of how and why test suites can become burdensome and shares solutions for keeping your test suites tidy. Ryan explores techniques for test parallelization and code restructuring that his company used to decrease the execution time of its test suites by more than 90 percent while more than tripling the number of tests. Take back new ways to fearlessly scale your agile testing.

1.
AT4
Session
6/6/2013 10:15 AM
"Scaling Your Tests:
Continued Change Without Fear"
Presented by:
Ryan Scott
Rally Software Development
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Ryan Scott
Rally Software Development
A software engineer and development manager at Rally Software, Ryan Scott scales teams,
production systems, and testing infrastructure. Before working for Rally, Ryan held software
engineer and technical architect roles for several large financial firms. He has software
engineering experience delivering software systems ranging from enormous data warehouses
to extensible JavaScript components. Ryan is passionate about keeping development teams
moving fast while making the right choices. He writes regularly about agile, software
engineering, and automated testing on the Rally Engineering blog. When not obsessing about
testing and scalability, Ryan enjoys playing with his kids, skiing, rock climbing, and home
brewing.

3. Scaling your tests
continued change without fear
Ryan Scott - Rally Software
rscott@rallydev.com

4. Who is this guy?
Ryan Scott
Rally Software
Development Manager
(fancy name for team lead)

5. Why do I care about this?
-A good test suite is a safety net
that enables fearless changes to a
codebase.
-As tests suites grow, they often
decay, becoming almost as much of a
burden as they once were a benefit.
-If your test suite takes longer
than it used to, you should care.

6. Tell me more.
-If you kick off your tests and go
play Wii*, you should care.
-If you don't run ALL of your tests
EVERY time you check in, you need
to pay attention.
-If you want to ship software
faster, you should care.

7. This doesn't sound
like it's for me!
-If you don't already have
automated tests, it's not for you.
-This is not an introduction to
automated testing.
-If you don't think this is the
right topic for you, I won't be
offended if you leave.

8. What do you know about this?
When I started at Rally:
-10 hours to run all tests
Now:
-build pipeline is "done" in 35
minutes now

9. 10 hours is a long time.
How many tests was that?
-5000 java tests
-1000 javascript tests
-2000 selenium browser tests
2009 Rally build pipeline:
1 hour:
commit
Java build
Java tests
Javascript tests
9 hours:
Browser tests

10. That sounds painful.
Why did you decide to change?
-Rapid growth caused major
scalability problems
-Older parts of the app were not
designed with testability in mind
-Existing browser tests not trusted
-"Fixes" caused more problems

11. Why did you continue?
-Move from 8 week to 1 week
releases
-Push for continuous delivery
-Can now release in 30 min if we
need to

12. This sounds too good to be
true.

13. Your cat pic is so 2009.
It didn't convince me.
I'm still skeptical.
-This really happened
-I couldn't find a meme for that

14. Ok, I'm sold.
What's the first step?
-Parallelize tests into 10 buckets
-Parallelization cut Rally's Java
tests from an hour to 6 min

15. Wait, you only saved 50
minutes out of ten hours!
-Give other test suites same
treatment
-Give your pipeline the same
treatment
-Parallelize data creation inside
of test suites

16. So what does your pipeline
look like now?
-11,000+ java tests
-3,000+ javascript tests
-3,600+ selenium browser tests
Java tests
Javascript
tests
commit
Java build
Browser
tests
???

17. That sounds too easy.
What's the catch?
-This is really hard
-Flaky tests
-Threading bugs
-Scalability issues

18. How long did it take to fix
all that?
-Ongoing effort
-Just scaling (not writing) tests
is probably 5-10% of our
engineering effort

19. That sounds painful and
expensive. Why do it?
-50 minutes of savings for each
developer multiple times per day
-That's just one of multiple test
suites
-That's real money

20. What about the other
test suites?
-We can automatedly regression test
the entire app in 35 minutes
-Focus on testing and quality lead
to crossing/diverging lines
-That's happy customers

21. What are crossing/diverging
lines?

22. Tell me about
flaky tests.
-Non deterministic failures
-Difficult to reproduce
-Data dependencies
-Test ordering issues
-Clicking too soon

23. What's the solution
for flaky tests?
-Logging for diagnostics
-Order independence
-Visual cues
-Understand asynchronicity

24. Tell me about
threading bugs in tests.
-What's not thread safe is shocking
-Statics/singletons
-Threading bugs in libraries
-Can be difficult to reproduce

25. Threading bugs sound a lot
like flaky tests.
-A failed assertion is not the only
way for a test to reveal a bug
-Not all threading problems are in
test code
-A failure is a failure - why make
a distinction?
-Flaky prod?

26. Tell me about
test scalability issues.
-Memory (and memory, and memory)
-IO (database, file system)
-Connection pool
-Not all test issues

27. I don't think you heard me.
This sounds REALLY expensive.
-It IS really expensive
-It is also really valuable
-Some teams will notice even more
value

28. I think I need to do this.
How do I mitigate the cost?
Finally a good question.

29. Ok, then answer it.
Do three things:
-Make it like production
-Dedicate resources
-Don't stop

30. How do I "make
it like production"?
-If you don't understand why a test
or a test suite failed, you have a
problem
-When things go wrong in prod, how
do you know?
-Why is prod different from your
test suite?

31. What should I be looking for?
-The same things as production:
-memory
-response time
-disk usage/IO
-error/exception rates
-cpu (JVM killquits)
-database activity
-Correlate obsessively (stoplight)

32. It seems like you could reuse
a bunch of the monitoring.
-A typical pattern is to push
monitoring down from production
-With parallelization, tests can
become a leading indicator for
production stability
-We've pushed monitoring from
browser tests to production

33. Why dedicate resources?
Everyone should writes tests.
-Everyone SHOULD write tests
-Not everyone is great at scaling
systems (that includes tests)
-Some problems are easier to solve
when you solve them regularly

34. It seems cheaper for devs to
do this in their spare time.
-This is a specialized skill set
-If everyone is doing it, no one is
doing it
-IF the tests are still trusted, a
stop the line event will result

35. Stop the line event?
-Term from lean manufacturing
-Huge build time slowdowns and
systemic instability will
inevitably stop the line
-The line should stop sooner
-Cyclical line stoppages

36. I'm totally sold.
What now?
-You forgot point #3
-DO NOT STOP
-I can't stress this enough

37. RANDOM BONUS MATERIAL
-Automatic thread dumps
-Thread dumps are amazing:
-DNS lookups
-Loading page announcements
-Lack of kernel entropy
-Memory cleanup test listeners

38. MORE RANDOM BONUS MATERIAL
-Request logs are also amazing:
-Dangling requests
-Missing clicks
-Long running requests
-On-demand build jobs let
developers leverage beefier
hardware