WORKS IN (X, Y, Z)Parallel Combination Testing With Selenium,JUnit, and Sauce LabsDavid Ross Drake
4/12/2013                                            2Dynacron Group• Surprisingly, not the evil corporation from that  80...
4/12/2013                                   3A Common Selenium Problem• Tests are slow• Lots of combinations need to be te...
4/12/2013                                       4Parameter Description                  Browser            -   Application...
4/12/2013                                   5Extant Solution: CSV• Simple for Manual    • Incapable of  Testers to Use    ...
4/12/2013            6   TYPESAFE CONFIG   Demo
But the Problem is Actually Worse…             IC   CP       IN       HI       EX       SB   CW   6C       PCRUS En       ...
4/12/2013                                            8How Many Combinations?• Browsers: A, B, C, D, E• Locales: 1, 2, 3, 4...
4/12/2013                                  9Each Once?• Browsers: A, B, C, D, E   1. (A, 1, X)• Locales: 1, 2, 3, 4       ...
4/12/2013                                               10All Pairs?• Browsers: A, B, C, D, E   1.  (A, 1, X)   11. (C, 3,...
4/12/2013                                               11All?• Browsers: A, B, C, D, E   1.  (A, 1, X)   11. (A, 4, Y)   ...
4/12/2013                                                12Risk vs RewardsRisk                           Rewardx = probabi...
4/12/2013                                                                       13Converting Combinations into Tests    Re...
4/12/2013                    14Parallelization• Easy to Implement   • See Sauce Labs blogs!• Very hard to do right   • See...
4/12/2013                                       15Class Browser Reuseb = time to start and close browsert = average time t...
4/12/2013                                       16Class Browser Reuse (small tests)15 = time to start and close browser6 =...
4/12/2013                                       17Class Browser Reuse (long tests)15 = time to start and close browser60 =...
4/12/2013                     18Tracking Parameterized andParallelized Tests• Descriptive method naming• Descriptive loggi...
4/12/2013                                      19Future Work• Remembering Tested Combinations in  Coverage Reports• Effect...
4/12/2013                                      20Questions?• Parallel Webtest: https://github.com/dynacron-  group/paralle...
Upcoming SlideShare
Loading in …5
×

Works in (X, Y, Z): Parallel Combination Testing With Selenium, JUnit, and Sauce

1,310 views

Published on

By David Drake, Lead Software Development Engineer in Test at Dynacron Group.

0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,310
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
21
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • Also, a great place to work.
  • Jump from here to demonstration of IHG’s site.
  • Limitations
  • Jump from here to demonstration of solution using typesafeconfig
  • Jump from here to demonstration of IHG’s site.
  • Jump from here to demonstration of IHG’s site.
  • Jump from here to demonstration of IHG’s site.
  • Jump from here to demonstration of IHG’s site.
  • Biggest problem: what the hell is x?
  • Hey: it takes forever to run all these combinations
  • Hey: it takes forever to run all these combinations
  • Hey: it takes forever to run all these combinations
  • Hey: it takes forever to run all these combinations
  • Hey: it takes forever to run all these combinations
  • Also, a great place to work.
  • Works in (X, Y, Z): Parallel Combination Testing With Selenium, JUnit, and Sauce

    1. 1. WORKS IN (X, Y, Z)Parallel Combination Testing With Selenium,JUnit, and Sauce LabsDavid Ross Drake
    2. 2. 4/12/2013 2Dynacron Group• Surprisingly, not the evil corporation from that 80’s dystopian sci-fi movie• Actually a Seattle consultancy that helps companies implement continuous delivery… • Integrated Testing • Custom Development • Big Data Strategies • Deployment Automation
    3. 3. 4/12/2013 3A Common Selenium Problem• Tests are slow• Lots of combinations need to be tested • Browsers • Versions • Operating Systems • Languages • Skins• Combination interfaces are very similar
    4. 4. 4/12/2013 4Parameter Description Browser - Application - Type - Version - OS Brand - Code - URL Component Locale - Language - Country
    5. 5. 4/12/2013 5Extant Solution: CSV• Simple for Manual • Incapable of Testers to Use containing• Allows some nested complicated data structures (PSV in • Hard to Read CSV…) • Ugly
    6. 6. 4/12/2013 6 TYPESAFE CONFIG Demo
    7. 7. But the Problem is Actually Worse… IC CP IN HI EX SB CW 6C PCRUS En X X X X X X X XUK En X X X X X XFrench X X X X X X X XGerman X X X X X XJapanese X X XChinese X X X X X X XKorean X X XItalian X X X XArabic XHebrew XPortuguese X XSpanish X X X X X X X XRussian XTurkish X
    8. 8. 4/12/2013 8How Many Combinations?• Browsers: A, B, C, D, E• Locales: 1, 2, 3, 4• Brands: X, Y, ZLet S1 be the size of the largest parameter array,S2 be the next largest, etc…
    9. 9. 4/12/2013 9Each Once?• Browsers: A, B, C, D, E 1. (A, 1, X)• Locales: 1, 2, 3, 4 2. (B, 2, Y)• Brands: X, Y, Z 3. (C, 3, Z) 4. (D, 4, X)Number of 5. (E, 1, Y)combinations: S1
    10. 10. 4/12/2013 10All Pairs?• Browsers: A, B, C, D, E 1. (A, 1, X) 11. (C, 3, Y) 2. (A, 2, Y) 12. (C, 4, Z)• Locales: 1, 2, 3, 4 3. (A, 3, Z) 13. (D, 1, X)• Brands: X, Y, Z 4. (A, 4, X) 14. (D, 2, Y) 5. (B, 1, Y) 15. (D, 3, Z) 6. (B, 2, Z) 16. (D, 4, X)Number of 7. (B, 3, X) 17. (E, 1, Y)combinations: (S1S2) 8. (B, 4, Y) 18. (E, 2, Z) 9. (C, 1, Z) 19. (E, 3, X) 10. (C, 2, X) 20. (E, 4, Y)
    11. 11. 4/12/2013 11All?• Browsers: A, B, C, D, E 1. (A, 1, X) 11. (A, 4, Y) 2. (A, 1, Y) 12. (A, 4, Z)• Locales: 1, 2, 3, 4 3. (A, 1, Z) 13. (B, 1, X)• Brands: X, Y, Z 4. (A, 2, X) 14. (B, 1, Y) 5. (A, 2, Y) 15. (B, 1, Z) 6. (A, 2, Z) 16. (B, 2, X)Number of 7. (A, 3, X) 17. (B, 2, Y)combinations: (S1S2S3) 8. (A, 3, Y) 18. (B, 2, Z) 9. (A, 3, Z) 19. (B, 3, X) 10. (A, 4, X) 20. …
    12. 12. 4/12/2013 12Risk vs RewardsRisk Rewardx = probability a given pair y = average time it takes tohas a failure only found test a combinationwhen together P = Fraction of testing timeR = probability that “each- taken testing “each-once”once” fails to reveal a as compared to “all-pairs”defect that could be seenin “all-pairs” M = S1 / S1S2R = (1 - x) ^ (S1S2 – S1)
    13. 13. 4/12/2013 13Converting Combinations into Tests Read Parameters • Add overrides for class, profiles, and system properties • Read as TypeSafe Config objects Form list of config combinations • Apply combination strategy from config Convert configurations back into json • Simple in TypeSafe Config – prevents writing custom object mapping Deserialize json combination in ParameterCombination object • Uses Jackson and @FromJson annotations as necessary
    14. 14. 4/12/2013 14Parallelization• Easy to Implement • See Sauce Labs blogs!• Very hard to do right • See demo
    15. 15. 4/12/2013 15Class Browser Reuseb = time to start and close browsert = average time to run a test methodm = average number of methods in a classF = fraction of time spent testing when usingone browser per class instead of per methodF = (b/m + t) / (b + t)
    16. 16. 4/12/2013 16Class Browser Reuse (small tests)15 = time to start and close browser6 = average time to run a test method5 = average number of methods in a classF = fraction of time spent testing when usingone browser per class instead of per methodF = (15/5 + 6) / (15 + 6) = 3/7 = 42.9%
    17. 17. 4/12/2013 17Class Browser Reuse (long tests)15 = time to start and close browser60 = average time to run a test method5 = average number of methods in a classF = fraction of time spent testing when usingone browser per class instead of per methodF = (15/5 + 60) / (15 + 60) = 21/25 = 84%
    18. 18. 4/12/2013 18Tracking Parameterized andParallelized Tests• Descriptive method naming• Descriptive logging• Demo!
    19. 19. 4/12/2013 19Future Work• Remembering Tested Combinations in Coverage Reports• Effectively displaying tested combinations• Converting to using Junit 4.11 style parameterized runner.
    20. 20. 4/12/2013 20Questions?• Parallel Webtest: https://github.com/dynacron- group/parallel-webtest• TypeSafe Config: https://github.com/typesafehub/config• Sauce post that provided the seed: http://sauceio.com/index.php/2010/10/paralle l-junit-4-and-selenium-part-three-parallelism- and-ondemand/

    ×