Your SlideShare is downloading. ×
0
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
QaOps - NYC Selenium Users Meetup - 03/15/12
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

QaOps - NYC Selenium Users Meetup - 03/15/12

2,261

Published on

NYC Metro Selenium Users Meetup 03/15/12 …

NYC Metro Selenium Users Meetup 03/15/12
http://www.meetup.com/NYCSelenium/events/52638682/

Dev + Ops = DevOps --> QA + Ops = QaOps

Chris Hoey, Senior IT Manager at Wireless Generation, will share some of the surprising ways Selenium has brought value to his company outside of the testing function.

Chris began his career at Wireless Generation as a Lead QA engineer responsible for writing many of the Selenium tests. When he transitioned over to Operations, he decided to carry the tests over to see new life not previously thought of.

This included periodically benchmarking applications from tests, helping customer care debug browser specific issues seen on their site and playing back the tests into a jmeter proxy to extend the tests into load testing scripts. It also spawned a new name (coined by Chris) to describe the joining of these two groups - QaOps.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,261
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Christopher Hoey ■ Senior IT Manager - Wireless Generation ■ Started in QA ■ Switched over to Operations ■ Selenium abuser since 2006 ■ Got to see the Selenium vs Webdriver Cage Fight liveDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 2. Thanks!Ashley WilsonLindsey BaronSauce LabsConductorNYC Metro Selenium UsersSelenium DevelopersJmeter Developers Founded 12/14/06 Closed 07/16/09 Dev + Ops = DevOps --> QA + Ops = QAOps NYC Metro Selenium Users Meetup March 15, 2012
  • 3. Show of hands Numbers collected and updated before sharing slides About 35 attended and participated in informal pollDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 4. Show of hands - primary job title Quality Assurance ? 12Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 5. Show of hands - primary job title Developer ? 10Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 6. Show of hands - primary job title Operations ? 5Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 7. Show of hands - experience with Selenium Have not used it? 5Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 8. Show of hands - experience with Selenium Tried Selenium IDE? 8Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 9. Show of hands - experience with Selenium Mastered Selenium IDE? 1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 10. Show of hands - experience with Selenium Tried Selenium RC/Webdriver? 13Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 11. Show of hands - experience with Selenium Mastered Selenium RC/Webdriver? 6Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 12. Show of hands - experience with Selenium Selenium tied to CI build system? 10Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 13. Show of hands - experience with Selenium Running a Selenium Grid? 7Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 14. Show of hands - use of Selenium Functional Testing New Features? 9Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 15. Show of hands - use of Selenium Regression Testing? 10Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 16. Show of hands - use of Selenium Load/Stress Testing? really? 1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 17. Show of hands - use of Selenium Benchmarking Performance? 1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 18. Show of hands - use of Selenium Automating Data Entry? 1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 19. Show of hands - use of Selenium Monitoring for Operations? 1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 20. Show of hands - use of Selenium Something Strange? 2 Filling out forms to renew trial licenses when expired Running build jobs for a build tool that does not have a cliDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 21. Dev + Ops = DevOps --> QA + Ops = QAOps Disclaimers Not a talk about best practices Forcing Selenium to do things not originally intended A lot of talk about Selenium IDE QAOps term kind of meant as a joke but might have meaning? Times have changed, tools improved and storms have passedDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 22. About Wireless Generation ● Create tools that make teachers jobs easier ● Primarily focused around assessment and reporting apps ● Mix of mature code/platforms and new development projects ● Experiencing rapid growth year over yearDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 23. Going MacGyver in early days of Ops Faith in monitors pretty low Manual configuration management Serious bugs in third party tools Large increases in user traffic Lots of new projects to juggle while keeping existing work online Sysadmins manually checking site =( Reuse anything that helps! Selenium to the rescue!Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 24. Suites of Smoke Tests for Operations ● Use Selenium IDE so anyone in Operations with Firefox can use ● Suites are happy path smoke tests that are not brittle ● Focus on touching all moving parts of infrastructure ● All suites can run with any username ● All suites can run on any environment ● Sysadmins running Selenium tests -- still done manually =/Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 25. Suites for Operations DEMO TIME Removing brittleness from Selenium IDE tests against a site with difficult locatorsDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 26. Suites for Operations - DEMO TIME All we really want to know is if we threw any errors while loading pages that touch on all parts of our platform. Two simple regex searches for every page gets us the desired minimal coverage Yes that looks way too simple but it did the job very well. These tests started finding issues before our customers did. Confidence increased dramatically!Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 27. Suites for Operations - DEMO TIME https://zzz.yyy.net/wgen/tpri/LoadTpriClasseSummary.do? ppkDefaultVersionId=7&ppkPeriodId=20&ppkClassId=550987013&ppkCompetencyDisplayId=1&ppkInstI d=33645685&ppkSchoolGroupId... We know the starting URL pattern. The rest of the URL is really dynamic and the page locators that happen to be available are not Selenium friendly. We could use XPath but that has proven to be really brittle for us. Solution - Store the html source and use storeEval with a regex to find the correct link every timeDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 28. Automation - judges.py ■ Headless Selenium servers ■ Suites of tests for each production webapp using smoke tests ■ Run through suites serially in a never ending loop ■ Any failure in suites marks the status as failed for that server ■ Look at each servers total result time running the suite based on thresholds ■ Running helps see what server is currently being tested ■ Results projected on wall in Ops areaDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 29. We have a fundamental problem Sysadmins CustomersDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 30. Solution - Any browser any platform Bromine GridDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 31. Loadtesting Selenium vs JMeter ● Lots of start up overhead ● Loads quickly ● ~30 concurrent sessions before ● ~300 concurrent sessions before strange things happen strange things happen ● Lots of requirements / coordination ● Java and a jar, headless is easy headless is a hackDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 32. Loadtesting with JMeter Queries to find interesting usersSelenium Scripts IN Scripts to replace variables for Selenium suites Playback Selenium scripts through JMeter proxy JMeter Scripts OUTDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 33. Loadtesting with JMeter DEMO TIME Setting up jmeter as a proxy recorder for Selenium playbackDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 34. Loadtesting with JMeter - DEMO TIME Start JMeter and create a new basic WorkBenchDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 35. Loadtesting with JMeter - DEMO TIME Add an HTTP Proxy Server: Edit -> Add -> Non-Test Elements -> HTTP Proxy ServerDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 36. Loadtesting with JMeter - DEMO TIME Set your port, any record filters of what to include or exclude and click start to fire up the proxyDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 37. Loadtesting with JMeter - DEMO TIME Set Firefox network settings to use localhost and the port you configured JMeter proxy to useDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 38. Loadtesting with JMeter - DEMO TIME Playback your tests using Selenium and watch as JMeter records the hitsDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 39. Loadtesting with JMeter - DEMO TIME Add a thread group by right clicking Test Plan -> Add -> Threads -> Thread GroupDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 40. Loadtesting with JMeter - DEMO TIME Name your thread group, pick how many threads you want, ramp up time and how many loopsDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 41. Loadtesting with JMeter - DEMO TIME Run your tests and set servers ablaze. Run -> StartDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 42. Finding hard bugs Uh oh! Just throwing more memory at the problem will not solve this!Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 43. Finding hard bugs Did we see this in the environments below Production? Not very noticeable unless looking hard for it.Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 44. Finding hard bugsRamping up JMeter threads of a specific user in an endless loop reproduces issue in every environmentDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 45. Dashboard Setup? Use Selenium to login and navigate to your various dashboards for TVs and sharing with teams.Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 46. QAOps? A lot of quick wins can be made here.Getting most of the focus these days.Certainly best of all worlds comingtogether. Takes a while to be a good Selenium IDE makes it trivialpractitioner based on tech debt | servers | for Ops to start running andtimeboxes | complexity | culture developing testsDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 47. Questions? We are hiring! http://www.wirelessgeneration.comDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 48. Picture Credits http://uploads.neatorama.com/wp-content/uploads/2010/06/macgyver.jpg http://www.packagingsuppliesbymail.com/images/C/Duct%20Tape.jpg http://www.russpainter.com/shop/images/clip.jpg http://g-ecx.images-amazon.com/images/G/01/stores/sport-goods/1209-huntsman.jpg http://www.agilegeoscience.com/storage/post-images/python-logo-small.png? __SQUARESPACE_CACHEVERSION=1316752792052Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 49. Picture Credits http://www.pptbackground.net/plog-content/images/powerpoint/education-backgrounds/raising-hands- together-powerpoint-backgrounds.jpg http://seleniumhq.org/images/selenium-ide-logo.png http://upload.wikimedia.org/wikipedia/en/0/07/Jakarta_jmeter_logo.jpg http://cdn.babble.com/momcrunch/files/2012/02/SauceLabs_Logo_Vert_150px.gif http://gigaom2.files.wordpress.com/2011/01/firefox-ie.pngDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 50. Picture Credits http://3.bp.blogspot.com/_t5Xqp0AYeg4/S74FdH-SO-I/AAAAAAAAAOU/OlMrI9m8VIE/s320/smileyface1.jpg http://blogs.babycenter.com/wp-content/uploads/2008/02/crying-face.jpg http://www.portlandmercury.com/imager/dr-strangelove/b/original/3791589/80b9/mwbw-drstrangelove.jpg http://stylreesta.files.wordpress.com/2011/07/caution_big.png http://en.wikipedia.org/wiki/File:Devops.pngDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012
  • 51. Picture Credits http://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Tux.png/220px-Tux.png http://www.wisedatasecurity.com/imagenes/ie-8.jpg http://seleniumhq.org/images/selenium-grid-logo.png http://www.studiosysadmins.com/site_media/attachments. gallery/2009/05/12/ServerRoomFireWeb.jpg http://blogs.splunk.com/wp-content/uploads/2010/02/Picture-65.pngDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users MeetupMarch 15, 2012

×