Christopher Hoey  ■ Senior IT Manager - Wireless Generation  ■ Started in QA  ■ Switched over to Operations  ■ Selenium ab...
Thanks!Ashley WilsonLindsey BaronSauce LabsConductorNYC Metro Selenium UsersSelenium DevelopersJmeter Developers          ...
Show of hands       Numbers collected and updated before sharing slides      About 35 attended and participated in informa...
Show of hands - primary job title  Quality Assurance ?           12Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Seleni...
Show of hands - primary job title                   Developer ?                      10Dev + Ops = DevOps --> QA + Ops = Q...
Show of hands - primary job title                  Operations ?                      5Dev + Ops = DevOps --> QA + Ops = QA...
Show of hands - experience with Selenium        Have not used it?                5Dev + Ops = DevOps --> QA + Ops = QAOpsN...
Show of hands - experience with Selenium             Tried Selenium                  IDE?                    8Dev + Ops = ...
Show of hands - experience with Selenium   Mastered Selenium         IDE?           1Dev + Ops = DevOps --> QA + Ops = QAO...
Show of hands - experience with Selenium            Tried Selenium            RC/Webdriver?                   13Dev + Ops ...
Show of hands - experience with Selenium   Mastered Selenium    RC/Webdriver?           6Dev + Ops = DevOps --> QA + Ops =...
Show of hands - experience with Selenium    Selenium tied to CI      build system?             10Dev + Ops = DevOps --> QA...
Show of hands - experience with Selenium              Running a            Selenium Grid?                   7Dev + Ops = D...
Show of hands - use of Selenium     Functional Testing      New Features?             9Dev + Ops = DevOps --> QA + Ops = Q...
Show of hands - use of Selenium                     Regression                      Testing?                        10Dev ...
Show of hands - use of Selenium                    Load/Stress                    Testing?              really?           ...
Show of hands - use of Selenium              Benchmarking              Performance?                    1Dev + Ops = DevOps...
Show of hands - use of Selenium         Automating Data             Entry?                1Dev + Ops = DevOps --> QA + Ops...
Show of hands - use of Selenium                Monitoring for                Operations?                      1Dev + Ops =...
Show of hands - use of Selenium        Something Strange?                                                2                ...
Dev + Ops = DevOps --> QA + Ops = QAOps                            Disclaimers                            Not a talk about...
About Wireless Generation   ●   Create tools that make teachers jobs easier   ●   Primarily focused around assessment and ...
Going MacGyver in early days of Ops Faith in monitors pretty low Manual configuration management Serious bugs in third par...
Suites of Smoke Tests for Operations  ●   Use Selenium IDE so anyone in Operations with Firefox can use  ●   Suites are ha...
Suites for Operations                               DEMO                               TIME                             Re...
Suites for Operations - DEMO TIME All we really want to know is if we threw any errors while loading pages that touch on a...
Suites for Operations - DEMO TIME https://zzz.yyy.net/wgen/tpri/LoadTpriClasseSummary.do? ppkDefaultVersionId=7&ppkPeriodI...
Automation - judges.py   ■   Headless Selenium servers   ■   Suites of tests for each production webapp using       smoke ...
We have a fundamental problem              Sysadmins                   CustomersDev + Ops = DevOps --> QA + Ops = QAOpsNYC...
Solution - Any browser any platform         Bromine         GridDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium ...
Loadtesting Selenium vs JMeter   ●   Lots of start up overhead             ●   Loads quickly   ●   ~30 concurrent sessions...
Loadtesting with JMeter                                   Queries to find                                  interesting use...
Loadtesting with JMeter                              DEMO                              TIME                             Se...
Loadtesting with JMeter - DEMO TIME                   Start JMeter and create a new basic WorkBenchDev + Ops = DevOps --> ...
Loadtesting with JMeter - DEMO TIME          Add an HTTP Proxy Server: Edit -> Add -> Non-Test Elements -> HTTP Proxy Serv...
Loadtesting with JMeter - DEMO TIME   Set your port, any record filters of what to include or exclude and click start to f...
Loadtesting with JMeter - DEMO TIME   Set Firefox network settings to use localhost and the port you configured JMeter pro...
Loadtesting with JMeter - DEMO TIME   Playback your tests using Selenium and watch as JMeter records the hitsDev + Ops = D...
Loadtesting with JMeter - DEMO TIME   Add a thread group by right clicking Test Plan -> Add -> Threads -> Thread GroupDev ...
Loadtesting with JMeter - DEMO TIME   Name your thread group, pick how many threads you want, ramp up time and how many lo...
Loadtesting with JMeter - DEMO TIME                       Run your tests and set servers ablaze. Run -> StartDev + Ops = D...
Finding hard bugs             Uh oh! Just throwing more memory at the problem will not solve this!Dev + Ops = DevOps --> Q...
Finding hard bugs Did we see this in the environments below Production? Not very noticeable unless looking hard for it.Dev...
Finding hard bugsRamping up JMeter threads of a specific user in an endless loop reproduces issue in every environmentDev ...
Dashboard Setup?  Use Selenium to login and navigate to your various dashboards for TVs and sharing with teams.Dev + Ops =...
QAOps?                                                       A lot of quick wins can                                      ...
Questions?              We are hiring!               http://www.wirelessgeneration.comDev + Ops = DevOps --> QA + Ops = QA...
Picture Credits            http://uploads.neatorama.com/wp-content/uploads/2010/06/macgyver.jpg             http://www.pac...
Picture Credits             http://www.pptbackground.net/plog-content/images/powerpoint/education-backgrounds/raising-hand...
Picture Credits            http://3.bp.blogspot.com/_t5Xqp0AYeg4/S74FdH-SO-I/AAAAAAAAAOU/OlMrI9m8VIE/s320/smileyface1.jpg ...
Picture Credits            http://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Tux.png/220px-Tux.png             http...
Upcoming SlideShare
Loading in...5
×

QaOps - NYC Selenium Users Meetup - 03/15/12

2,285

Published on

NYC Metro Selenium Users Meetup 03/15/12
http://www.meetup.com/NYCSelenium/events/52638682/

Dev + Ops = DevOps --> QA + Ops = QaOps

Chris Hoey, Senior IT Manager at Wireless Generation, will share some of the surprising ways Selenium has brought value to his company outside of the testing function.

Chris began his career at Wireless Generation as a Lead QA engineer responsible for writing many of the Selenium tests. When he transitioned over to Operations, he decided to carry the tests over to see new life not previously thought of.

This included periodically benchmarking applications from tests, helping customer care debug browser specific issues seen on their site and playing back the tests into a jmeter proxy to extend the tests into load testing scripts. It also spawned a new name (coined by Chris) to describe the joining of these two groups - QaOps.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,285
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

QaOps - NYC Selenium Users Meetup - 03/15/12

  1. 1. Christopher Hoey ■ Senior IT Manager - Wireless Generation ■ Started in QA ■ Switched over to Operations ■ Selenium abuser since 2006 ■ Got to see the Selenium vs Webdriver Cage Fight liveDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  2. 2. Thanks!Ashley WilsonLindsey BaronSauce LabsConductorNYC Metro Selenium UsersSelenium DevelopersJmeter Developers Founded 12/14/06 Closed 07/16/09 Dev + Ops = DevOps --> QA + Ops = QAOps NYC Metro Selenium Users Meetup March 15, 2012
  3. 3. Show of hands Numbers collected and updated before sharing slides About 35 attended and participated in informal pollDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  4. 4. Show of hands - primary job title Quality Assurance ? 12Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  5. 5. Show of hands - primary job title Developer ? 10Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  6. 6. Show of hands - primary job title Operations ? 5Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  7. 7. Show of hands - experience with Selenium Have not used it? 5Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  8. 8. Show of hands - experience with Selenium Tried Selenium IDE? 8Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  9. 9. Show of hands - experience with Selenium Mastered Selenium IDE? 1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  10. 10. Show of hands - experience with Selenium Tried Selenium RC/Webdriver? 13Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  11. 11. Show of hands - experience with Selenium Mastered Selenium RC/Webdriver? 6Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  12. 12. Show of hands - experience with Selenium Selenium tied to CI build system? 10Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  13. 13. Show of hands - experience with Selenium Running a Selenium Grid? 7Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  14. 14. Show of hands - use of Selenium Functional Testing New Features? 9Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  15. 15. Show of hands - use of Selenium Regression Testing? 10Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  16. 16. Show of hands - use of Selenium Load/Stress Testing? really? 1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  17. 17. Show of hands - use of Selenium Benchmarking Performance? 1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  18. 18. Show of hands - use of Selenium Automating Data Entry? 1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  19. 19. Show of hands - use of Selenium Monitoring for Operations? 1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  20. 20. Show of hands - use of Selenium Something Strange? 2 Filling out forms to renew trial licenses when expired Running build jobs for a build tool that does not have a cliDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  21. 21. Dev + Ops = DevOps --> QA + Ops = QAOps Disclaimers Not a talk about best practices Forcing Selenium to do things not originally intended A lot of talk about Selenium IDE QAOps term kind of meant as a joke but might have meaning? Times have changed, tools improved and storms have passedDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  22. 22. About Wireless Generation ● Create tools that make teachers jobs easier ● Primarily focused around assessment and reporting apps ● Mix of mature code/platforms and new development projects ● Experiencing rapid growth year over yearDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  23. 23. Going MacGyver in early days of Ops Faith in monitors pretty low Manual configuration management Serious bugs in third party tools Large increases in user traffic Lots of new projects to juggle while keeping existing work online Sysadmins manually checking site =( Reuse anything that helps! Selenium to the rescue!Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  24. 24. Suites of Smoke Tests for Operations ● Use Selenium IDE so anyone in Operations with Firefox can use ● Suites are happy path smoke tests that are not brittle ● Focus on touching all moving parts of infrastructure ● All suites can run with any username ● All suites can run on any environment ● Sysadmins running Selenium tests -- still done manually =/Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  25. 25. Suites for Operations DEMO TIME Removing brittleness from Selenium IDE tests against a site with difficult locatorsDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  26. 26. Suites for Operations - DEMO TIME All we really want to know is if we threw any errors while loading pages that touch on all parts of our platform. Two simple regex searches for every page gets us the desired minimal coverage Yes that looks way too simple but it did the job very well. These tests started finding issues before our customers did. Confidence increased dramatically!Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  27. 27. Suites for Operations - DEMO TIME https://zzz.yyy.net/wgen/tpri/LoadTpriClasseSummary.do? ppkDefaultVersionId=7&ppkPeriodId=20&ppkClassId=550987013&ppkCompetencyDisplayId=1&ppkInstI d=33645685&ppkSchoolGroupId... We know the starting URL pattern. The rest of the URL is really dynamic and the page locators that happen to be available are not Selenium friendly. We could use XPath but that has proven to be really brittle for us. Solution - Store the html source and use storeEval with a regex to find the correct link every timeDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  28. 28. Automation - judges.py ■ Headless Selenium servers ■ Suites of tests for each production webapp using smoke tests ■ Run through suites serially in a never ending loop ■ Any failure in suites marks the status as failed for that server ■ Look at each servers total result time running the suite based on thresholds ■ Running helps see what server is currently being tested ■ Results projected on wall in Ops areaDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  29. 29. We have a fundamental problem Sysadmins CustomersDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  30. 30. Solution - Any browser any platform Bromine GridDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  31. 31. Loadtesting Selenium vs JMeter ● Lots of start up overhead ● Loads quickly ● ~30 concurrent sessions before ● ~300 concurrent sessions before strange things happen strange things happen ● Lots of requirements / coordination ● Java and a jar, headless is easy headless is a hackDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  32. 32. Loadtesting with JMeter Queries to find interesting usersSelenium Scripts IN Scripts to replace variables for Selenium suites Playback Selenium scripts through JMeter proxy JMeter Scripts OUTDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  33. 33. Loadtesting with JMeter DEMO TIME Setting up jmeter as a proxy recorder for Selenium playbackDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  34. 34. Loadtesting with JMeter - DEMO TIME Start JMeter and create a new basic WorkBenchDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  35. 35. Loadtesting with JMeter - DEMO TIME Add an HTTP Proxy Server: Edit -> Add -> Non-Test Elements -> HTTP Proxy ServerDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  36. 36. Loadtesting with JMeter - DEMO TIME Set your port, any record filters of what to include or exclude and click start to fire up the proxyDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  37. 37. Loadtesting with JMeter - DEMO TIME Set Firefox network settings to use localhost and the port you configured JMeter proxy to useDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  38. 38. Loadtesting with JMeter - DEMO TIME Playback your tests using Selenium and watch as JMeter records the hitsDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  39. 39. Loadtesting with JMeter - DEMO TIME Add a thread group by right clicking Test Plan -> Add -> Threads -> Thread GroupDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  40. 40. Loadtesting with JMeter - DEMO TIME Name your thread group, pick how many threads you want, ramp up time and how many loopsDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  41. 41. Loadtesting with JMeter - DEMO TIME Run your tests and set servers ablaze. Run -> StartDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  42. 42. Finding hard bugs Uh oh! Just throwing more memory at the problem will not solve this!Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  43. 43. Finding hard bugs Did we see this in the environments below Production? Not very noticeable unless looking hard for it.Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  44. 44. Finding hard bugsRamping up JMeter threads of a specific user in an endless loop reproduces issue in every environmentDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  45. 45. Dashboard Setup? Use Selenium to login and navigate to your various dashboards for TVs and sharing with teams.Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  46. 46. QAOps? A lot of quick wins can be made here.Getting most of the focus these days.Certainly best of all worlds comingtogether. Takes a while to be a good Selenium IDE makes it trivialpractitioner based on tech debt | servers | for Ops to start running andtimeboxes | complexity | culture developing testsDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  47. 47. Questions? We are hiring! http://www.wirelessgeneration.comDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  48. 48. Picture Credits http://uploads.neatorama.com/wp-content/uploads/2010/06/macgyver.jpg http://www.packagingsuppliesbymail.com/images/C/Duct%20Tape.jpg http://www.russpainter.com/shop/images/clip.jpg http://g-ecx.images-amazon.com/images/G/01/stores/sport-goods/1209-huntsman.jpg http://www.agilegeoscience.com/storage/post-images/python-logo-small.png? __SQUARESPACE_CACHEVERSION=1316752792052Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  49. 49. Picture Credits http://www.pptbackground.net/plog-content/images/powerpoint/education-backgrounds/raising-hands- together-powerpoint-backgrounds.jpg http://seleniumhq.org/images/selenium-ide-logo.png http://upload.wikimedia.org/wikipedia/en/0/07/Jakarta_jmeter_logo.jpg http://cdn.babble.com/momcrunch/files/2012/02/SauceLabs_Logo_Vert_150px.gif http://gigaom2.files.wordpress.com/2011/01/firefox-ie.pngDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  50. 50. Picture Credits http://3.bp.blogspot.com/_t5Xqp0AYeg4/S74FdH-SO-I/AAAAAAAAAOU/OlMrI9m8VIE/s320/smileyface1.jpg http://blogs.babycenter.com/wp-content/uploads/2008/02/crying-face.jpg http://www.portlandmercury.com/imager/dr-strangelove/b/original/3791589/80b9/mwbw-drstrangelove.jpg http://stylreesta.files.wordpress.com/2011/07/caution_big.png http://en.wikipedia.org/wiki/File:Devops.pngDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  51. 51. Picture Credits http://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Tux.png/220px-Tux.png http://www.wisedatasecurity.com/imagenes/ie-8.jpg http://seleniumhq.org/images/selenium-grid-logo.png http://www.studiosysadmins.com/site_media/attachments. gallery/2009/05/12/ServerRoomFireWeb.jpg http://blogs.splunk.com/wp-content/uploads/2010/02/Picture-65.pngDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012

×