ebay @ Hasso Plattner Institut Potsdam

Testing @ eBay International

Michael Palotas
Francois Reynaud
Potsdam, 20.01.2012

The Company

2
Produced by: Michael Palotas

eBay Inc.

3

Setup

Mobile
Global Portland Global
Product USA Product

San Jose Chennai
USA India

EU
Quality
Engineering
Global Local
Product
(Zürich, London,
Berlin, Paris)
Product

Shanghai Zürich,
China London

Adjacencies
(i.e. Brands
for Friends)
4

Not only the Web

Windows
Desktop
Web
Mobile

5

Testing at eBay in the past

6

Testing Today – Ownership of Quality

Manual Test

Consulting
+
Coaching

Automation

7

Manual Test

Still very important, mainly exploratory type

All new features are tested manually first

Edge cases

Not automatable tests

Manual testing still yields the highest number of bugs

8

Automation

Automation is not about eliminating jobs !!!

Save time and reinvest into manual testing

Regression Testing (fully) automated

Programmatic approach (JAVA)

Open source only (WebDriver/Selenium 2 GRID)

3500 end to end testcases running in less than 1 hour
9

GRID

eBay Europe:
-  3500 e2e regression testcases
-  2-3 minutes execution time (each)

executions
Execution time 7000 minutes (~116 hours = ~5 days)
Sequential

Parallel
Execution Test
1
Test
2
Test
3
Test
…
Test
…
Test
3500

Execution Time

Parallel Test Test Test Test Test
Parallel executions

150 Virtual Machines / Browsers in
execution in Test Test
the WebDriver GRID
Test Test Test
Execution time: 60 minutes
Grid
Test Test Test Test Test

Execution Time

10

Coaching / Consulting

EU team conducts project audits

Improvement initiatives

Consulting for dev and test teams

11

Project Audits

12

EU Quality Engineering Team – Automation Impact

Team % of global Bugs 2009
"   Team
makes up for 20.00%
18.00% Start of
only 2% of global test 16.00% automation
efforts in
engineer headcount 14.00%
early 2009 Contribution
12.00% Trendline
10.00%
8.00%
6.00%
"   Clear correlation 4.00%
between global bug 2.00% headcount share

contribution and .00% 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13

Week
automation
investment / coverage 18.0% Team % of global Bugs 2010
16.0%
14.0%
12.0%
10.0%
8.0%
6.0%
4.0%
headcount share
2.0% headcount share
0.0% 13
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38Palotas 43 44 45 46 47 48 49 50 51
Produced by: Michael 39 40 41 42
Week

Bug Quality

Bug Quality = real bugs / submitted bugs
100
Bug Quality in % (2008-2010)

95

90
Start of
automation
efforts
85 Bug Quality 2008-2010

80

75
Q1 08 Q2 08 Q3 08 Q4 08 Q1 09 Q2 09 Q3 09 Q4 09 Q1 10 Q2 10 Q3 10 Q4 10

14

INNOVATE…and talk about it

Sydney, Australia, October 5th 2011

Selenium GRID has 1.5K downloads / day


Mobile

Delight our customers with a fantastic user experience


The problem


Where to test?


Manual Testing & Automation


Obstacles and opportunities

Mobile Web is easy…

Native App is not…

Planning to work with Google on open source project “nativeDriver”:
http://code.google.com/p/nativedriver/


Selenium tool suite integration at eBay

HPI - 2012

Overview

What is Selenium
The different tools under the selenium umbrella
Integrating selenium for web testing
Leveraging the protocol for other technologies
Conclusion


What is Selenium today ?

Web testing tool
Highly successful open source project, supported by big companies
Stable, and fast evolving

Protocol describing the user interactions
Comes with most browsers implementations
Works with most common languages


What it's not

A test framework
A network testing/monitoring tool
A performance testing tool


History

Started as a pure JS implementation
•  1 implementation for all browsers
•  browser specific features had to be hacks
•  XSS issues

Webdriver merged to Selenium
•  real events
•  new, documented, API

•  backward compatibility of sel1 protocol
•  faster,easier to maintain
•  1 implementation per browser


Webdriver JSON protocol

The one page to read :
http://code.google.com/p/selenium/wiki/JsonWireProtocol
The core of the product
Built to be a standard
Based on the HTTP protocol
Specifies :
•  all the “normal” browsing interactions:
GET /session/:sessionid/url
POST /session/:sessionid/url
•  and the new generation interactions :
POST /session/:sessionid/touch/longclick


The vision behind selenium

The cure :)
Vendors are responsible for the implementation
Protocol becomes the standard for web testing


When to use selenium

Reproduce use cases
Unit test of the web components for x-browser validation
End to end validation / regression ( eBay EU )


When not to use it

Everything else :
•  Performance testing
•  Driver for anything with a UI...


Component 0/3 : Selenium IDE

Firefox plugin
Super easy to start
Usually first interaction with selenium


Component 1/3 : Selenium client driver

What testers see as “selenium”
Doesn't do much : generates the API calls
The part that will be called :
•  from the test framework
•  from the CI
Official selenium support
•  Java, c#, python, ruby
Not Officially supported
•  Perl, php


Component 2/3 : Selenium server

The more interesting part
Where the contributors spend their times
Drives the AUT
Officially supported by the selenium team:
•  IE, Firefox,iphone,android
Supported by vendors:
•  Chrome,Opera


Component 3/3 : Selenium grid

Proxy the HTTP requests to do 3 things :
•  Load balancer
•  Environment manager
•  Disaster recovery


Writing Selenium tests

IDE
•  Record / replay
•  (too) easy
•  Selenese / html
Client / server
•  Code
•  No record and replay
•  Testing requires basic dev skills


Selenium IDE - SIDE

Record and replay design
Entry barrier very low
Maintenance is a nightmare
Building abstraction layers for SIDE = reinventing the wheel


Selenium client driver

Clear separation on the test code and the driver code
Stable API : update the driver without impacting the clients
Allows to run remotely
The tests are just another piece of software


Tests are code !

Spend time on the design
Use page object
Don't test everything


Next step : scaling using grid

Scalability :
•  manage multiple browsers / OS installations
•  run tests in parallel
•  hooks to deploy everything in a production environment


Grid architecture


Grid – create plugins !

Extension points
Change the default behavior :
•  Registration
•  Test
•  Command
Or tune it :
•  Custom priority
•  Custom matching

Plugins will :
•  execute on the hub
•  access the internal hub state ( mutable )
•  allow to overwrite methods


Plugin example : eBay

We moved away from Grid1
•  Self healing features covered mostly connectivity issues
•  Nodes were taken out after a crash, but no automatic recovery
•  We needed to manage more than just the browser : the OS too.
•  Managing a growing grid is error prone


eBay grid requirements

Maintenance :
•  (re)start everything automatically
•  Adding 1 node = 1 line in a config file
•  Nodes are restarted before having issues
•  Test isolation
Debugging :
•  Showcase for grid , adding features without changing the tests : VNC


Default grid startup overview

Launcher
Hub
registration request webdriver
server

node


Grid : logic of the eBay plugin

The plugin ( proxy ) we currently use implement 2 interfaces :
•  Registration listener
Restores the VM to a clean state
Starts the display and VNC
Starts webdriver server
Validation
•  Test session listener :
Counts the tests
Directs the browsers to the right display


Integration with virtualization

Hub •  Custom node proxy
•  Registration doesn’t come from the node
•  New proxy code in the classpath. No need to re-compile grid.
•  Extra params for the proxy


Console


VNC


More integration : TWIN

http://code.google.com/p/twin

•  Webdriver implementation for automating Windows graphical applications.

• Open source project from eBay


eBay example : TWIN for Turbo Lister 2


Challenges of Turbo lister2 Automation

On top of driving the application, the test infrastructure needs to handle :
•  the software installation
•  the config files

This can be handled by a plain grid hub by adding a plugin :
•  When a node registers :
it has a vm ID property
the hub check for existing snapshots
the hub triggers the installation for each site
TurboLister2 is started for the first time
the hub create a snapshot
•  When a new test is started :
a VM is assigned
the hub revert to the appropriate snapshot


Advantages of a single grid

All the tests follow the same high level life cycle :
•  automatically manage the context around the test
•  run the tests until the OS / AUT crashes
•  try to recover
•  or restart
Products release at different times :
•  re-assign the VMs


Next step : mobile

Native driver + link
Early stage
Android
•  working for basic features
•  Linux makes it easy to integrate

Iphone
•  working for basic features
•  Mac OS makes it harder to integrate


Conclusion

Not the silver bullet
It allows your QE team to :
•  use the language they're comfortable to write their test
•  reuse the helpers
•  clean API allows you to extends to new technology “easily”


Resources

https://github.com/freynaud/grid-spine-common
•  Depends on selenium 2.4.0
•  VMWare integration + restart every N tests logic
https://github.com/freynaud/grid-spine-selenium
•  Depends on common
•  All the selenium specific logic, VNC integration, Linux vs Windows code.

Built using
•  noVNC : http://kanaka.github.com/noVNC/
•  tightVNC
•  VIX ( VMWare API ) http://labs.vmware.com/flings/vix-java-toolkit
•  VMWare WS http://vijava.sourceforge.net/


ebay @ Hasso Plattner Institut Potsdam

More Related Content

What's hot

Similar to ebay @ Hasso Plattner Institut Potsdam

More from Michael Palotas

Recently uploaded

ebay @ Hasso Plattner Institut Potsdam