Testing @ eBay International            Michael Palotas            Francois Reynaud            Potsdam, 20.01.2012
The Company                                             2              Produced by: Michael Palotas
eBay Inc.                                           3            Produced by: Michael Palotas
Setup                 Mobile   Global         Portland           Global   Product         USA               Product   San ...
Not only the Web                        Windows                        Desktop               Web                     Mobil...
Testing at eBay in the past                                                             6                              Pro...
Testing Today – Ownership of Quality                     Manual Test         Consulting             +         Coaching    ...
Manual Test        Still very important, mainly exploratory type        All new features are tested manually first        ...
Automation       Automation is not about eliminating jobs !!!       Save time and reinvest into manual testing            ...
GRIDeBay Europe:-  3500 e2e regression testcases-  2-3 minutes execution time (each)                           executions ...
Coaching / Consulting             EU team conducts project audits                  Improvement initiatives             Con...
Project Audits                                                12                 Produced by: Michael Palotas
EU Quality Engineering Team – Automation Impact                                                                          T...
Bug QualityBug Quality = real bugs / submitted bugs       100                                    Bug Quality in % (2008-20...
INNOVATE…and talk about it                 Sydney, Australia, October 5th 2011               Selenium GRID has 1.5K downlo...
Mobile         Delight our customers with a fantastic user experience                                                     ...
The problem              Produced by: Michael Palotas
Where to test?                 Produced by: Michael Palotas
Manual Testing & Automation                              Produced by: Michael Palotas
Obstacles and opportunitiesMobile Web is easy…Native App is not…Planning to work with Google on open source project “nativ...
Selenium tool suite integration at eBay             HPI - 2012
OverviewWhat is SeleniumThe different tools under the selenium umbrellaIntegrating selenium for web testingLeveraging the ...
What is Selenium today ?Web testing toolHighly successful open source project, supported by big companiesStable, and fast ...
What its notA test frameworkA network testing/monitoring toolA performance testing tool                                   ...
HistoryStarted as a pure JS implementation•  1 implementation for all browsers•  browser specific features had to be hacks...
Webdriver JSON protocolThe one page to read :    http://code.google.com/p/selenium/wiki/JsonWireProtocolThe core of the pr...
The vision behind seleniumThe cure :)Vendors are responsible for the implementationProtocol becomes the standard for web t...
When to use seleniumReproduce use casesUnit test of the web components for x-browser validationEnd to end validation / reg...
When not to use itEverything else :•  Performance testing•  Driver for anything with a UI...                              ...
Component 0/3 : Selenium IDEFirefox pluginSuper easy to startUsually first interaction with selenium                      ...
Component 1/3 : Selenium client driverWhat testers see as “selenium”Doesnt do much : generates the API callsThe part that ...
Component 2/3 : Selenium serverThe more interesting partWhere the contributors spend their timesDrives the AUTOfficially s...
Component 3/3 : Selenium gridProxy the HTTP requests to do 3 things :•  Load balancer•  Environment manager•  Disaster rec...
Writing Selenium testsIDE•  Record / replay•  (too) easy•  Selenese / htmlClient / server•  Code•  No record and replay•  ...
Selenium IDE - SIDERecord and replay designEntry barrier very lowMaintenance is a nightmareBuilding abstraction layers for...
Selenium client driverClear separation on the test code and the driver codeStable API : update the driver without impactin...
Tests are code !Spend time on the designUse page objectDont test everything                           Produced by: Michael...
Next step : scaling using gridScalability :•  manage multiple browsers / OS installations•  run tests in parallel•  hooks ...
Grid architecture                    Produced by: Michael Palotas
Grid – create plugins !Extension pointsChange the default behavior :•  Registration•  Test•  CommandOr tune it :•  Custom ...
Plugin example : eBayWe moved away from Grid1•  Self healing features covered mostly connectivity issues•  Nodes were take...
eBay grid requirementsMaintenance :•  (re)start everything automatically•  Adding 1 node = 1 line in a config file•  Nodes...
Default grid startup overview                                         Launcher     Hub                  registration reque...
Grid : logic of the eBay pluginThe plugin ( proxy ) we currently use implement 2 interfaces :•  Registration listener     ...
Integration with virtualizationHub                           •  Custom node proxy                              •  Registra...
Console          Produced by: Michael Palotas
VNC      Produced by: Michael Palotas
More integration : TWIN                                            http://code.google.com/p/twin•  Webdriver implementatio...
eBay example : TWIN for Turbo Lister 2                                         Produced by: Michael Palotas
Challenges of Turbo lister2 AutomationOn top of driving the application, the test infrastructure needs to handle :•  the s...
Advantages of a single gridAll the tests follow the same high level life cycle :•  automatically manage the context around...
Next step : mobileNative driver + linkEarly stageAndroid•  working for basic features•  Linux makes it easy to integrateIp...
ConclusionNot the silver bulletIt allows your QE team to :•  use the language theyre comfortable to write their test•  reu...
Resourceshttps://github.com/freynaud/grid-spine-common•  Depends on selenium 2.4.0•  VMWare integration + restart every N ...
Upcoming SlideShare
Loading in …5
×

ebay @ Hasso Plattner Institut Potsdam

524 views

Published on

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
524
On SlideShare
0
From Embeds
0
Number of Embeds
18
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

ebay @ Hasso Plattner Institut Potsdam

  1. 1. Testing @ eBay International Michael Palotas Francois Reynaud Potsdam, 20.01.2012
  2. 2. The Company 2 Produced by: Michael Palotas
  3. 3. eBay Inc. 3 Produced by: Michael Palotas
  4. 4. Setup Mobile Global Portland Global Product USA Product San Jose Chennai USA India EU Quality Engineering Global Local Product (Zürich, London, Berlin, Paris) Product Shanghai Zürich, China London Adjacencies (i.e. Brands for Friends) 4 Produced by: Michael Palotas
  5. 5. Not only the Web Windows Desktop Web Mobile 5 Produced by: Michael Palotas
  6. 6. Testing at eBay in the past 6 Produced by: Michael Palotas
  7. 7. Testing Today – Ownership of Quality Manual Test Consulting + Coaching Automation 7 Produced by: Michael Palotas
  8. 8. Manual Test Still very important, mainly exploratory type All new features are tested manually first Edge cases Not automatable tests Manual testing still yields the highest number of bugs 8 Produced by: Michael Palotas
  9. 9. Automation Automation is not about eliminating jobs !!! Save time and reinvest into manual testing Regression Testing (fully) automated Programmatic approach (JAVA) Open source only (WebDriver/Selenium 2 GRID) 3500 end to end testcases running in less than 1 hour 9 Produced by: Michael Palotas
  10. 10. GRIDeBay Europe:-  3500 e2e regression testcases-  2-3 minutes execution time (each) executions Execution time 7000 minutes (~116 hours = ~5 days)Sequential ParallelExecution Test 1 Test 2 Test 3 Test … Test … Test 3500 Execution TimeParallel Test Test Test Test Test Parallel executions 150 Virtual Machines / Browsers inexecution in Test Test the WebDriver GRID Test Test Test Execution time: 60 minutesGrid Test Test Test Test Test Execution Time 10 Produced by: Michael Palotas
  11. 11. Coaching / Consulting EU team conducts project audits Improvement initiatives Consulting for dev and test teams 11 Produced by: Michael Palotas
  12. 12. Project Audits 12 Produced by: Michael Palotas
  13. 13. EU Quality Engineering Team – Automation Impact Team % of global Bugs 2009"   Team makes up for 20.00% 18.00% Start ofonly 2% of global test 16.00% automation efforts inengineer headcount 14.00% early 2009 Contribution 12.00% Trendline 10.00% 8.00% 6.00%"   Clear correlation 4.00%between global bug 2.00% headcount sharecontribution and .00% 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13 Weekautomationinvestment / coverage 18.0% Team % of global Bugs 2010 16.0% 14.0% 12.0% 10.0% 8.0% 6.0% 4.0% headcount share 2.0% headcount share 0.0% 13 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38Palotas 43 44 45 46 47 48 49 50 51 Produced by: Michael 39 40 41 42 Week
  14. 14. Bug QualityBug Quality = real bugs / submitted bugs 100 Bug Quality in % (2008-2010) 95 90 Start of automation efforts 85 Bug Quality 2008-2010 80 75 Q1 08 Q2 08 Q3 08 Q4 08 Q1 09 Q2 09 Q3 09 Q4 09 Q1 10 Q2 10 Q3 10 Q4 10 14 Produced by: Michael Palotas
  15. 15. INNOVATE…and talk about it Sydney, Australia, October 5th 2011 Selenium GRID has 1.5K downloads / day Produced by: Michael Palotas
  16. 16. Mobile Delight our customers with a fantastic user experience Produced by: Michael Palotas
  17. 17. The problem Produced by: Michael Palotas
  18. 18. Where to test? Produced by: Michael Palotas
  19. 19. Manual Testing & Automation Produced by: Michael Palotas
  20. 20. Obstacles and opportunitiesMobile Web is easy…Native App is not…Planning to work with Google on open source project “nativeDriver”:http://code.google.com/p/nativedriver/ Produced by: Michael Palotas
  21. 21. Selenium tool suite integration at eBay HPI - 2012
  22. 22. OverviewWhat is SeleniumThe different tools under the selenium umbrellaIntegrating selenium for web testingLeveraging the protocol for other technologiesConclusion Produced by: Michael Palotas
  23. 23. What is Selenium today ?Web testing toolHighly successful open source project, supported by big companiesStable, and fast evolvingProtocol describing the user interactionsComes with most browsers implementationsWorks with most common languages Produced by: Michael Palotas
  24. 24. What its notA test frameworkA network testing/monitoring toolA performance testing tool Produced by: Michael Palotas
  25. 25. HistoryStarted as a pure JS implementation•  1 implementation for all browsers•  browser specific features had to be hacks•  XSS issuesWebdriver merged to Selenium •  real events •  new, documented, API •  backward compatibility of sel1 protocol •  faster,easier to maintain •  1 implementation per browser Produced by: Michael Palotas
  26. 26. Webdriver JSON protocolThe one page to read : http://code.google.com/p/selenium/wiki/JsonWireProtocolThe core of the productBuilt to be a standardBased on the HTTP protocolSpecifies :•  all the “normal” browsing interactions: GET /session/:sessionid/url POST /session/:sessionid/url•  and the new generation interactions : POST /session/:sessionid/touch/longclick Produced by: Michael Palotas
  27. 27. The vision behind seleniumThe cure :)Vendors are responsible for the implementationProtocol becomes the standard for web testing Produced by: Michael Palotas
  28. 28. When to use seleniumReproduce use casesUnit test of the web components for x-browser validationEnd to end validation / regression ( eBay EU ) Produced by: Michael Palotas
  29. 29. When not to use itEverything else :•  Performance testing•  Driver for anything with a UI... Produced by: Michael Palotas
  30. 30. Component 0/3 : Selenium IDEFirefox pluginSuper easy to startUsually first interaction with selenium Produced by: Michael Palotas
  31. 31. Component 1/3 : Selenium client driverWhat testers see as “selenium”Doesnt do much : generates the API callsThe part that will be called :•  from the test framework•  from the CIOfficial selenium support•  Java, c#, python, rubyNot Officially supported•  Perl, php Produced by: Michael Palotas
  32. 32. Component 2/3 : Selenium serverThe more interesting partWhere the contributors spend their timesDrives the AUTOfficially supported by the selenium team:•  IE, Firefox,iphone,androidSupported by vendors:•  Chrome,Opera Produced by: Michael Palotas
  33. 33. Component 3/3 : Selenium gridProxy the HTTP requests to do 3 things :•  Load balancer•  Environment manager•  Disaster recovery Produced by: Michael Palotas
  34. 34. Writing Selenium testsIDE•  Record / replay•  (too) easy•  Selenese / htmlClient / server•  Code•  No record and replay•  Testing requires basic dev skills Produced by: Michael Palotas
  35. 35. Selenium IDE - SIDERecord and replay designEntry barrier very lowMaintenance is a nightmareBuilding abstraction layers for SIDE = reinventing the wheel Produced by: Michael Palotas
  36. 36. Selenium client driverClear separation on the test code and the driver codeStable API : update the driver without impacting the clientsAllows to run remotelyThe tests are just another piece of software Produced by: Michael Palotas
  37. 37. Tests are code !Spend time on the designUse page objectDont test everything Produced by: Michael Palotas
  38. 38. Next step : scaling using gridScalability :•  manage multiple browsers / OS installations•  run tests in parallel•  hooks to deploy everything in a production environment Produced by: Michael Palotas
  39. 39. Grid architecture Produced by: Michael Palotas
  40. 40. Grid – create plugins !Extension pointsChange the default behavior :•  Registration•  Test•  CommandOr tune it :•  Custom priority•  Custom matchingPlugins will :•  execute on the hub•  access the internal hub state ( mutable )•  allow to overwrite methods Produced by: Michael Palotas
  41. 41. Plugin example : eBayWe moved away from Grid1•  Self healing features covered mostly connectivity issues•  Nodes were taken out after a crash, but no automatic recovery•  We needed to manage more than just the browser : the OS too.•  Managing a growing grid is error prone Produced by: Michael Palotas
  42. 42. eBay grid requirementsMaintenance :•  (re)start everything automatically•  Adding 1 node = 1 line in a config file•  Nodes are restarted before having issues•  Test isolationDebugging :•  Showcase for grid , adding features without changing the tests : VNC Produced by: Michael Palotas
  43. 43. Default grid startup overview Launcher Hub registration request webdriver server node Produced by: Michael Palotas
  44. 44. Grid : logic of the eBay pluginThe plugin ( proxy ) we currently use implement 2 interfaces :•  Registration listener Restores the VM to a clean state Starts the display and VNC Starts webdriver server Validation•  Test session listener : Counts the tests Directs the browsers to the right display Produced by: Michael Palotas
  45. 45. Integration with virtualizationHub •  Custom node proxy •  Registration doesn’t come from the node •  New proxy code in the classpath. No need to re-compile grid. •  Extra params for the proxy Produced by: Michael Palotas
  46. 46. Console Produced by: Michael Palotas
  47. 47. VNC Produced by: Michael Palotas
  48. 48. More integration : TWIN http://code.google.com/p/twin•  Webdriver implementation for automating Windows graphical applications.• Open source project from eBay Produced by: Michael Palotas
  49. 49. eBay example : TWIN for Turbo Lister 2 Produced by: Michael Palotas
  50. 50. Challenges of Turbo lister2 AutomationOn top of driving the application, the test infrastructure needs to handle :•  the software installation•  the config filesThis can be handled by a plain grid hub by adding a plugin :•  When a node registers : it has a vm ID property the hub check for existing snapshots the hub triggers the installation for each site TurboLister2 is started for the first time the hub create a snapshot•  When a new test is started : a VM is assigned the hub revert to the appropriate snapshot Produced by: Michael Palotas
  51. 51. Advantages of a single gridAll the tests follow the same high level life cycle :•  automatically manage the context around the test•  run the tests until the OS / AUT crashes•  try to recover•  or restartProducts release at different times :•  re-assign the VMs Produced by: Michael Palotas
  52. 52. Next step : mobileNative driver + linkEarly stageAndroid•  working for basic features•  Linux makes it easy to integrateIphone•  working for basic features•  Mac OS makes it harder to integrate Produced by: Michael Palotas
  53. 53. ConclusionNot the silver bulletIt allows your QE team to :•  use the language theyre comfortable to write their test•  reuse the helpers•  clean API allows you to extends to new technology “easily” Produced by: Michael Palotas
  54. 54. Resourceshttps://github.com/freynaud/grid-spine-common•  Depends on selenium 2.4.0•  VMWare integration + restart every N tests logichttps://github.com/freynaud/grid-spine-selenium•  Depends on common•  All the selenium specific logic, VNC integration, Linux vs Windows code.Built using•  noVNC : http://kanaka.github.com/noVNC/•  tightVNC•  VIX ( VMWare API ) http://labs.vmware.com/flings/vix-java-toolkit•  VMWare WS http://vijava.sourceforge.net/ Produced by: Michael Palotas

×