This is a presentation I made for the Kraków Java User Group on test automation and how to solve the challenges around it to make it really useful for development teams. It contains some examples of how we are doing it at Akamai's Web department, and some based on my own experience.

1. Creating Testing Tools to Support Development
Chema del Barco
SDET Manager, Akamai Technologies

2. Warming up
How many of you…
… Work as dedicated testers?
… Create automated tests?
… Think you have enough tests?
… Rely on automated tests from testers?
… Trust these automated tests?

3. Warming up
How do we use test automation at Akamai?
• Different products & systems
• Java, JS, Ruby, Python…
• A lot of backend, less frontend
• Part of workflow
• We also do manual testing
• Lots of challenges

4. Challenges of test automation
Challenges of Test Automation

5. Challenges of test automation
1. Not testing the right thing

6. Not testing the right thing
Real Life Example:
Volvo’s Automatic Collision Avoidance
(2010)

7. Not testing the right thing
https://www.youtube.com/watch?v=aNi17YLnZpg

8. Not testing the right thing - lessons learned
1. Everybody makes mistakes
2. There is no such thing as 100% test coverage
3. There are always risks involved in any delivery
4. Testing can tell us where these risks are
5. We can mitigate risks if we know them

9. Not testing the right thing
Some failed demos later (2014)…

10. Not testing the right thing
https://www.youtube.com/watch?v=kWiwS-43xpk

11. Not testing the right thing
How can we mitigate this?

12. Not testing the right thing
Don’t lose the user’s point of view
PO/Testers can use
BDD / Spec by Example

13. Not testing the right thing
Developers
can use TDD

14. Not testing the right thing
Or you can all work
together to use ATDD!

15. Not testing the right thing
Alternative approach:
Use Domain Abstractions
(ex: Actors & Actions)

16. Challenges of test automation
2. The Test Environment Hell

17. The Test Environment Hell
• Complex applications tend to rely on big Integrated
Environments for validating changes
• Environments are so complex that require constant
troubleshooting and maintenance
• Testers (and builds) are often blocked and teams
spend a lot of time investigating if problem is app or
environment related

18. Build UT
IT ST Delivery
Build Validate
Dev
Stream
SDET
Stream
UT – Unit Testing Promote – Deploy to Next
CT – Component Testing Test Environment
IT – Integration Testing Push – Manual code push
ST – System Testing
SIT – System Integration Testing
Promote
Push
DEV Environment
INT Environment
Push
Ex: Akamai’s Luna Control Center test environments

19. Build UT
IT ST Delivery
Build Validate
Dev
Stream
SDET
Stream
UT – Unit Testing Promote – Deploy to Next
CT – Component Testing Test Environment
IT – Integration Testing Push – Manual code push
ST – System Testing
SIT – System Integration Testing
Promote
Push
DEV Environment
INT Environment
Push
Ex: Akamai’s Luna Control Center test environments

20. The Test Environment Hell
How can we mitigate this?

21. 1. Test the right thing in the right environment
“Separation of Concerns”:
1. Test that your change works, in isolation (UT)
2. Test that your app works with your change, in
isolation (CT / IT)
3. Test that your app works with your change, in
integration (ST)
4. Test that your system works in integration (SIT)

22. 1. Test the right thing in the right environment
• Isolated Component (Service) Testing
• Exclusive for Team
• MockServer to mock external
dependencies (provided by the
service owners)
• SUT thinks it’s in production
• Can test also Integration (Contract)
• Very stable, maintained by
devs/testers
• Testers can automate and validate
faster
• Everyone is less frustrated
Service
Under
Test (SUT)
Service 2
(MockServer)
Service 4
(MockServer)
Service 3
(MockServer)
Local Test Env.

23. Build UT CT
IT ST SIT Delivery
Build Validate
Dev
Stream
SDET
Stream
Promote
Push
Promote
DEV Environments
INT Environment SYS Environment
Push
QA Environment
1. Test the right thing in the right environment
If a test fails here it we know for
a fact that it will be because of:
1. The environment
2. A service not following its API
contract, or
3. An actual integration bug
(+ IT + ST)

24. 2. Implement automatic retry-on-error techniques
Some test frameworks like TestNG allow for several ways
to automatically retry failed tests, by:
1. Running “<test-outputs>testng-failed.xml” after a run with failed
tests [Good]
2. Run TestNG programmatically and implement a “retry test”
transformer to the @Test annotation [Great!]
3. If you don’t want to retry all, you can create a @Retry annotation
to implement the “retry test” listener [Great!]

25. 2. Implement automatic retry-on-error techniques (option 2)

26. 2. Implement automatic retry-on-error techniques (option 2)

27. 2. Implement automatic retry-on-error techniques (option 2)

28. 3. Use plugins in your CI to detect flaky tests
https://wiki.jenkins-ci.org/display/JENKINS/Test+Results+Analyzer+Plugin

29. Challenges of test automation
3. Doing test automation for the
wrong reasons

30. 1. “I don’t have to test anything manually”
Manual Testing
is CRUCIAL in
Agile because
machines cannot
think outside the
box …Yet ;)
“Agile Testing: A Practical Guide for Testers and Agile Teams”, Lisa Crispin & Janet Gregory

31. 2. Testing as a separated work flow
Build UT
IT/ST OKBuild
DEV Environments
QA Environment
Dev
Stream
Test
Stream
FAIL
“Ready
to Test”
Development “DONE”
Tester waits for
changes to be
deployed and runs
tests
Development “STARTS”
Tester provides FEEDBACK

32. Not understanding WHY we need test automation
Build UT CT
IT ST SIT Delivery
Build Validate
Promote
Push
Promote
DEV Environments
INT Environment SYS Environment
Push
QA Environment
Build
Pipeline
Test
Stream
The goal of test automation is to provide FEEDBACK
Test Automation should be another form of delivery

33. Not understanding WHY we need test automation
• Feedback MUST be:
• Reliable (no false positives/negatives)
• Fast (as early as possible)
• Scalable (keeps being fast when growing)
• Runnable by anyone
• It’s a WHOLE TEAM thing

34. Challenges of test automation
4. Building Vs. Using

35. Building vs. Using
• Now that testers are toolsmiths, they can also be
infected by the ”building-everything-from-scratch”
disease
• There is a tool for pretty much anything. If there is not,
search again (at least you should find a starting point)
• A good tester should always try to find the right tool
for the right kind of testing

36. ● In 2014 we had to send 10k+ HTML emails to our users
● HTML emails are inconsistently rendered by different email clients:
• Some do not support HTML at all,
• Some do not render it consistently with W3C specifications
● You never know which email client your users are using
• Desktop clients: Outlook 2002, 2013, Thunderbird
• Web clients: GMail (in Chrome, FF, IE), Yahoo! Mail, etc
● You cannot easily automate if email looks good, but you can automate
rendering in ~10 mail clients
Example

37. Example (cont)

38. Building Test Frameworks
How can a Test Framework be USEFUL?

39. 1. Easy to Write Tests
when().
get("/store").
then().
body("store.book.findAll { it.price < 10 }.title",
hasItems("Sayings of the Century",
”Moby Dick"));
when().
get("/store").
then().
body("store.book.author.collect { it.length()}.sum()",
greaterThan(50));
library: RestAssured
https://github.com/rest-assured/rest-assured

40. 2. Reporting
library: cucumber-reporting
https://github.com/damianszczepanik/cucumber-reporting

41. 2. Reporting
library: logging-selenium
http://loggingselenium.sourceforge.net/usage.html

42. 3. Debugging
Library: curl-logger
(Maciej Gawinecki)
http://nomoretesting.logdown.com/
https://github.com/dzieciou/curl-logger
curl 'http://google.com/' -H 'Accept: */*' -H
'Content-Length: 0' -H 'Host: google.com'
-H 'Connection: Keep-Alive' -H 'User-
Agent: Apache-HttpClient/4.5.1
(Java/1.8.0_45)' --compressed --insecure
--verbose

43. 4. Transparent X-Platform & X-Browser Testing
Libraries:
WebDriver
http://www.seleniumhq.org
appium
https://github.com/appium/appium

44. 5. Project Management Tool Integration
Zephyr API (ZAPI)
http://docs.getzephyr.apiary.io

45. Challenges of test automation
5. Thinking that end-to-end test
automation solves everything

46. Typical scenario of relying in end-to-end test automation

47. Example: the pain of a real delivery pipeline
Days Left Pass
%
Notes
1 5% Everything is broken! Signing in to the service is broken... Almost all
tests sign in a user so almost all tests failed.
0 4% A partner team we rely on deployed a bad build to their testing
environment yesterday.
-1 54% A dev broke the save scenario yesterday (or the day before?). Half the
tests save a document at some point in time. Devs spent most of the
day determining if it's a frontend bug or a backend bug
-2 54% It's a frontend bug, devs spent half of today figuring out where
-3 54% A bad fix was checked in yesterday. The mistake was pretty easy to
spot, though, and a correct fix was checked in today.
http://googletesting.blogspot.com/2015/04/just-say-no-to-more-end-to-end-tests.html

48. Example: the pain of a real delivery pipeline
Days
Left
Pass % Notes
-4 1% Hardware failures occurred in the lab for our testing environment.
-5 84% Many small bugs hiding behind the big bugs (e.g., sign-in broken, save
broken). Still working on the small bugs.
-6 87% We should be above 90%, but are not for some reason.
-7 89.54% (Rounds up to 90%, close enough.) No fixes were checked in
yesterday, so the tests must have been flaky yesterday
http://googletesting.blogspot.com/2015/04/just-say-no-to-more-end-to-end-tests.html

49. ©2016 AKAMAI | FASTER FORWARDTM
Problems with end-to-end tests
● Long
• Developers need to wait long for feedback about their changes
● Flaky
• Sensitive to environment and subsystems failures, timeouts, etc.
• Reduce the developer's trust in test, as a result flaky tests are often ignored
● Hard to isolate root cause
• Developers need to find the specific lines of code causing the bug
• For >1M LOC it's like trying to find a needle in a haystack.
http://googletesting.blogspot.com/2015/04/just-say-no-to-more-end-to-end-tests.html

50. ©2016 AKAMAI | FASTER FORWARDTM
How to address it? Do end-to-end only if necessary!
Maintenance
Slower tests
Flakiness
Move Fast & Don't Break Things, GTAC 2014

51. ©2016 AKAMAI | FASTER FORWARDTM
Cooling down
Summary

52. ©2016 AKAMAI | FASTER FORWARDTM
Summary
• Test Automation is a form of development and
should be treated as such
• It also suffers the same problems
• Think of WHY you need it before doing it
• There are plenty of tools and libraries to make it
more useful

53. Thank You!
Feel free to send questions to jdelbarc@akamai.com