The document compares agile QA approaches like end-to-end testing and discusses how Quid uses both Selenium and RainforestQA for testing. It notes that while Selenium allows for fast automated testing, it is tied to the frontend implementation. RainforestQA uses human testers which is easier to maintain but slower. For complex products like Quid, RainforestQA has scaled testing better and freed up engineer time for more exploratory testing.

1. COMPARING
AGILE QA APPROACHES
TO END-TO-ENDTESTING
Daria Mehra, Director of QE at Quid
Talk at Lean StartupWeek, 11/03/16

2. Why test at all?
■ Can’t we write bug-free software?
– Industry Average: "about 15 - 50 errors per 1000 lines of delivered code.”
(Steve McConnell, Code Complete)
■ Can’t we fix the bugs as the users report them?
– Are you Facebook?
■ Cost of fixing the bug goes up with time from inception
– Prevent it at no cost*, fix it pre-release for $, fix it post-release for $$$$.
■ “Hope is not a strategy.” (Google SRE)

3. Buggy math
■ Mercedes recently ran an ad saying that its new car has 100 million lines of code.That’s
impressive—until you realize that on average, any piece of software has a few bugs per
thousand lines of code, so the car probably has thousands of vulnerabilities.
– Chris Dixon of Andreessen Horowitz, replying to Obama’s 6 tech challenges
■ 100 mln loc * 15 bugs/kloc => 1.5 mln bugs!
■ Ford F-150 pickup, 150 mln loc => 2.25 mln bugs
■ Assume 0.001% probability of bug in a given month
■ Get 1 bug per working day…

4. How to move fast
■ Remember “Move fast and break things”?
■ Slogan v2: “Move fast and build things”!
■ Because you don’t move faster with a broken leg.
■ Quality vsVelocity is a false dichotomy
■ Quality improves as waste is removed,
leading to velocity
– Kaizen: “change for the better”
– What Agile was called in 1986

5. Bimodal IT: where do you land?
http://techbeacon.com/gartners-bimodal-it-mistake-devops-can-deliver-velocity-quality
Y u so slow?
Don’t do that “Fast” won’t last
Awesome. How?

6. Path to quality
Velocity: High.
Except for when it breaks.
Quality: ?????
Velocity: High.
Quality: High.
https://danashby.co.uk/2016/10/19/continuous-testing-in-devops/

7. Who needs end-to-end tests?
■ http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/
■ https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html
■ End-to-end testing considered harmful. If that’s all you do. (Steve Smith)
■ Unit tests are great! At Quid: 5000+ unit tests, 108 e2e user-workflow tests.

8. 2 unit tests, 0 integration tests
If this video won’t play in the slide deck, watch it here:
http://thisotplife.tumblr.com/post/137415092034/2-unit-tests-0-integration-tests

9. needs end-to-end tests!
■ What’s Quid?
– Quid is a platform that searches, analyzes and visualizes
the world’s collective intelligence to help
answer strategic questions
■ Huge data set + upload your own
– Data-dependent execution
■ Search -> NLP -> knowledge network -> viz
■ Rich UI with many workflows:
– Lots of ways to slice and dice the data
■ Import and export and browsers, oh my!
■ Must test important user workflows daily in CI

10. Two roads diverged
■ Manual-only testing is not a feasible path: slow, expensive, unreliable.
■ Two ways I’ve done it: full automation with Selenium, and RainforestQA.

11. Selenium approach
■ Automated end-to-end testcases in Python / Ruby / etc,
executed by driving browser actions via Selenium framework
■ Often with a layer of structured natural language on top (Gherkin or Cucumber)
■ Pros:
– Fast execution -> quick feedback loop
– Runs in CI on developer branches and master -> always releasable master
■ Cons:
– Tied to the front-end implementation, your CSS is now your API
– Finicky, tends to suffer from timing issues with page state
– Hard to debug failures, is this a product regression or a test issue?
“Element is not clickable at point (x, y). Other element would receive the click: …”
– Misses regressions that human would notice
– Takes engineering skill to add and maintain testcases

12. Release process w/Selenium
UnitTests + FunctionalTests + End-to-End SeleniumTests

13. RainforestQA approach
■ Automated end-to-end testcases in unstructured natural language (English),
executed by crowdsourced human testers via RainforestQA service
■ Pros:
– Easy to add and maintain testcases, non-engineers can do it
– Easy to debug test failures (screenshot + human comment)
– No dependence on front-end implementation, only user-visible workflow
– Human eye
– Parallel scalable execution
■ Cons:
– Slower test execution at human speed
– Annual contract expense
– Runs on staging and/or production, not on dev branches

14. Release process w/RainforestQA
UnitTests + FunctionalTests End-to-End RainforestQATests

15. Why we picked RainforestQA
■ Not either/or:
– Quid uses Selenium in production monitoring checks
■ Scaling out Rainforest testcases was SO MUCH EASIER
■ Solved regression testing for a complex product
■ Nightly testing is close enough to CI/CD for us
– Next up: early access testing with feature flags
■ Allows QE team to stay small
■ Promotes collaboration on quality between PM, CS, QE and Dev
■ Frees up Quality Engineer time for “artisanal” exploratory testing
■ … And that has made all the difference.

16. Ask me anything
■ Daria Mehra, Director of QE at Quid
■ Look us up: www.quid.com
■ Email me: dmehra@quid.com
■ Read my post on quality process: http://tinyurl.com/dmehra-ikea
■ Read about how I spend my days: http://tinyurl.com/dmehra-day
■ And we ARE hiring!