In this presentation we explain how we use Watir, Ruby, Cumcumber and other supporting technologies to allow end to end testing in MyHeritage.
These are the links to resource mentioned in the presentation:
Ruby - https://www.ruby-lang.org/en/
Watir - http://watirwebdriver.com/
page-object - https://github.com/cheezy/page-object
Selenium Grid - https://github.com/SeleniumHQ/selenium/wiki/Grid2
Selenium-Grid-Extras - https://github.com/groupon/Selenium-Grid-Extras
Jenkins - https://jenkins-ci.org/
We also explain how QA automation engineers are an integral part of the Continuous Deployment process at MyHeritage
Boost PC performance: How more available memory can improve productivity
MyHeritage - End 2 End testing Infra
1. Matan Goren and Yehuda Miller
QA Engineers
Test Automation in a
Continuous Deployment
Environment
2. • More than 80 million registered users
• 28 million family trees
• Website, mobile apps (iOS and Android),
desktop apps (windows and Mac)
• 2.5 billion profiles (names in family trees)
• 4.5 billion historical documents and records.
Including the world’s largest collection of
newspapers
• 200 million photos
• 42 languages
• 215 employees
Who we are
3. Where we were
• All manual QA
• Repetitive tasks
• Time consuming
• Two service packs a week
• Holding back development
• SPC size limits error handling
4. • Number of tests
• Desktop: 538 (469)
• Mobile-web 50 (0)
• Runtime of full suite: 37 min (51 min)
• RND size
• Number of testers: 13 (7 automations)
• Developers: 50
• Grids: 4 (1)
• Tests run on prod and semi staging env.
• Continuous Deployment
Where we are today:
5. Gray Area for full image
“You don't have to be a genius or a visionary or
even a college graduate to be successful. You
just need a framework and a dream.”
- Michael Dell
7. • Easy language for inexperienced
developers
• Open source community is very active
• It ain’t Java!
Ruby
8. • Native to Ruby
• Cleaner than Selenium
• Adds methods not native to Selenium
• “Being able to select an element by a explicit
identifier” –watirmelon.com
Watir-webdriver
10. • Provides a simple interface to the
define and interact with elements on a
page.
• Works with both Watir and Selenium.
• Simple way of introducing OOP
without a lot of technical knowledge
required
• Page factory module handles common
page classes action.
Page-Object gem
11. • BDD tool
• Tests are written in plain English
• Makes for easy debugging
• Easy move from Manual test to automatic test
• Helps soften the introduction to
programming
• “Top down” programming
• Write a step
• Define the step
• Make it work
• Pre and post test hooks
Cucumber.io
15. • Homegrown tool
• Takes a given suite and distributes the
test across several groups
• Utilizes open-source code from
cucumber and parallel_tests gems
• Multiple reports problem is solved by a
report merger tool that was developed
in house.
Grid Grouper
16. • Chuck Norris!
• Was already in use by RND
• Open source (lot’s of available plugins)
• Schedule automated builds
• Kick off manual builds with a few clicks
• Control build parameters (server, suite, etc).
• Can build in any of our environments with any
combination of tags
• Holds build history for a specified amount of
time
• Merged report is stored in each build.
• HipChat and email notifications
Jenkins
20. • Comprised of Selenium Grid and
Selenium-Grid-Extras
• Selenium-Grid-Extras makes the basic
setup streamlined
• Gives extra features such as:
• Pulling in updated drivers for all browsers
• Also for Selenium itself
The Grids
21. • We have 4 grids
• Each grid has 4 VMs
• The VMs include 1 hub-node and 3 other nodes
• The hubs are Jenkins slaves
• Each node can run 4 parallel tests on Chrome
• One grid is dedicated to CD
• We can pull a grid “offline” for testing infra
upgrades
• Ability to have multiple test runs in parallel on
different environments and tags
• Only runs through Jenkins
• Jenkins selects grids automatically based on availability
and run history
How we use the grids
22. “How long does it take you to get one line of
code into production?”
- Wisdom of the Internet ;)
23. • We moved to CD in the last 6 months
• On average we distribute code to
production 20 times per day.
• Comes with challenges
• Automated test on every code release
• Only minimal tests on each release
• Dev is responsible to run appropriate
tests PRE-RELEASE in staging
The CD mindset
24. • Risk reduction to production
• Small incremental changes are easier to
monitor and revert in necessary
• Has lower impact on the system
• Increasing R&D velocity
• Avoiding wasted time on merges and complex
coordination before dist.
• Allow R&D and product to experiment
and innovate more frequently.
RND Goals
25. • Provide a safety net
• Avoid the bottleneck of manual sanity
testing
• Reliability
• No flaky tests
• No false failures
• QA E2E test should not be the new
bottleneck
QA Goals
26. • QA are responsible for E2E tests only.
• Dev write and maintain Unit tests
• E2E tests are written during
development process
• All QA members monitor CD suite
results.
• CD E2E tests are NOT acceptance
tests
Responsibilities
27. • CD flow from commit to prod is ~25
minutes
• Current QA CD suite takes 1.5 minute
in best case scenario.
• Runs in parallel to some of the jobs in
the CD flow
• Limited by time frame of parallel jobs
• Still have room for further growth
Timing
28. • Minimal suite – No full regression on
commit
• Occasional false failures due to
env/site performance/human error
Risks
29. • Reliability
• Velocity of tests
• Pinpointing the failure
• CD halt on failure.
• Large scale tests refactoring due to
changes.
Challenges of CD
30. “A challenge only becomes an obstacle
when you bow to it.”
- Ray A. Davis
31. • Maintaining production data integrity
• Reliance on pre-existing data
• Running on local staging env
• Fast adaptation to feature flags
• Identifying failure reasons
• Need a true BDD/TDD mindset
Challenges QA faces
32. • Testing on more browsers
• IE
• Spartan
• Firefox
• Suites for our native apps
• iOS
• Android
• Upgrading the infra
• Gems, grids, Ruby…
The road ahead…
Number in parenthesis is last year
Desktop tests are on Chrome only for now.
Mobile-web tests are running in chrome with emulation mode turned on.
Full suite runs at least 2x daily
Semi staging is production database with offline code.
Anyone can write/read this (even devs/product)
Easy translation of a manual test to automated test
Multiple scenarios in one feature
Feature description
Scenario description
Highlighted parts are regexps that are used as arguments in the step def
Need to balance what is added to a scenario (don’t over-test, but don’t leave it too sparse)
Tags on the top determine the test suit
“on” is a method that initializes a page class.
We use Page-object class to represent an actual page
Top section is definition of elements in the page.
Have the ability to add custom elements and attributes (support for Angular, etc.)
Second section is methods performed in the page.
Loaded? Method is used for verification that a page loaded.
We keep the verification method of the page itself in the class of that page
Because our setup is very unique, we needed to build this tool
Working on putting additional effort to publish it as a gem
Running a failed test up to 3 time. Considered failed only if fails three times.Each run has it’s own report.
Failed step is marked.Includes stack trace
Screenshot of failure is embedded. Open with the link.
Allows multiple people to run tests at the same time on different environments
We started CD only with unit tests. Added QA tests within two weeks because the site went down several times which only our tests could have caught
During development process, it’s the dev and QA’s responsibility to run automations the new code.
In release cycles this quickly it’s impossible to test everything manually or run a large automation suite
E2E test run twice: Pre-dist on a Canary server (production server without traffic), and post-dist on prod after web servers get the code
Pre-dist is only one which fails the flow, post dist still requires immediate investigation. (All QA monitor this)
Note: prefer reliability and speed of tests over coverage and safety to enable us to distribute more frequently. Reduce dev time on the actual commit/dist.
Adds responsibility to devs to make for damn sure their code works
If one of our tests fail it stops the build
Can’t commit test code that is not synched with prod.