eBay’s European quality engineering team has broad experience implementing end-to-end test automation in different software development environments (agile, waterfall, co located , outsourced, distributed). This presentation illustrates the key lessons learned from a technical and business perspective: 1. Flip the testing triangle 2. Everyone knows what’s automated 3. A tool is not a strategy 4. Write the right tests 5. Instant Feedback is essential 6. Automation is software development itself 7. Maintainability is king 8. Speak the same language as the developers 9. Invest into the test Infrastructure 10. Manual Testing is still very important

1. Test Automation –
10 (sometimes painful) Lessons learned
Michael Palotas & Dominik Dary
Quality Engineering Team Europe
eBay International
April 2012

2. eBay Inc. Facts
Founded in 1995
Based in San Jose, California
27770 Employees worldwide

3. Platform Facts
104 million
active buyers and sellers worldwide
68 Billion USD
transaction volume 2011
2000 USD
transaction volume every second

4. Data Facts
9 petabytes of data
In our Hadoop and Teradata clusters
2 billion page views
each day
75 billion database calls
each day

5. Mobile Facts
70,000,000 times
eBay Inc’s apps have been downloaded
5 billion USD
transacted 2011 on mobile devices
1 million items
are listed with using mobile apps every week

6. Who we are
Michael Palotas
Head of Quality Engineering Europe
E-mail: mpalotas@ebay.com
Dominik Dary
Staff Software Engineer in Test
E-mail: ddary@ebay.com

7. Lesson 1: Flip the testing triangle
Manual
Tests
Manual Tests
E2E
Tests
E2E Automation
Tests Integration Tests
Unit
Tests Unit Tests
Inspiration: http://patrickwilsonwelsh.com/?p=32

8. Lesson 2: Everyone needs to know what is
automated
Use Test Aspects** Advantages
• Test aspects briefly describe • Clear overview for all team
“why” and “what” has to be members about test coverage
tested, without describing • Simplicity helps you to do
“how” quick reviews and
• Enables for early testing walkthroughs with your product
because tests can be written at owner
the same time the user story is • Enables you to identify manual
created and automated test duplicates
• Test aspects are structurally • Traceability between test
derived cases and specification
**Swiss Testing Day Presentation 2010: “Mit Testaspekten früh, intelligent und redundanzfrei testen” Melanie Späth
Image Source: http://www.masternewmedia.org/images/social_software_impact_individual_organizations.jpg

9. Lesson 3: A Tool is not a Strategy
A test tool/testing framework won't
teach your testers how to test
A clear testing process is mandatory
for test automation
Test Aspects and modeling of the
Biz-Domain layer is the key success
factor for your test strategy

10. Lesson 4: Write the right tests
Source: http://farm1.static.flickr.com/119/269335955_03a0ff4b67_o.jpg

11. Lesson 4: Write the right tests
Automate the business critical use
cases of your customer
Focus on end-to-end testing rather
than UI component tests
Design testability into your
product

12. Lesson 5: Instant Feedback is essential
Development Test
D T D T D T D T D T D T D T D T

13. Lesson 6: Automation is software development
Skillset & Knowledge Engineering practice
• You need people with software • Task oriented development
development skills and an helps you to get things done
appetite for testing • Follow the basic rules like lean
• Automation mindset is essential development principles
• Good understanding of the • Design the test before coding it
architecture of the application • Do code reviews
under test is required • Your developers can give you
very good feedback about your
code quality

14. Lesson 7: Maintainability is king
Source: http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef01348602ac19970c-pi

15. Lesson 7: Maintainability is king
Technical Debt Lean Test Automation
• Activities that result in • Eliminate Waste Definition of Done
technical debt include • Build Quality In Reviews
• Postponing
documentation and
• Create Knowledge Document the essentials
writing tests • Defer Commitment Don’t build a framework
• attending to TODO • Deliver Fast
comments and
• tackling compiler and • Respect People Long Leash
static code analysis • Optimize the Whole
warnings
• Knowledge that isn't
shared around the
organization
• Code that is too confusing
to be modified easily

16. Lesson 8: Speak the same language as the
developers
Use the same programming
language as development
Developers can write or maintain
tests if needed
Enables for a cross functional team
where everyone can work on nearly
every task

17. Lesson 9: Invest into the test infrastructure
Speed up and simplify
the creation of test
data
Offer and use web
services
Web based test data
creation tools can be
used in test auto-
mation and manual
testing

18. Lesson 10: Manual testing is still very important
Automated tests only do what you have implemented:
at the same speed, same order, with exactly the same
mouse clicks
Manual testing allows you to go off the beaten path
Test automation in combination with exploratory
testing combine the best of both worlds
Quality = Manual + Automated Testing

19. Conclusion of our ten lessons learned
1. Flip the testing triangle 6. Automation is software
2. A tool is not a strategy development itself
3. Everyone knows 7. Maintainability is king
what’s automated 8. Speak the same
4. Write the right tests language as the
5. Instant Feedback is developers
essential 9. Invest into the test
Infrastructure
10.Manual testing is still
very important

20. Thank You!
Questions or Comments?