Reaching for Your Quality Stretch Goals: Testing at Realtor.com

© 2015 Move, Inc. All rights reserved. Do not copy or distribute.
Reaching your Quality Stretch
@ realtor.com
Klaus Salchner
VP Engineering
https://www.linkedin.com/in/ksalchner
@ksalchner
April/5, 2017

© 2016 Move, Inc. All rights reserved. Do not copy or distribute. 2
Agenda
• A/B Testing to measure customer impact – 5-6 min
• Group activity – 5-6 min
• Where in the stack to invest in test automation – 5-6 min
• Testing for reliability, resilience und recovery – 5-6 min
• Q&A – 5-6 min

A/B Testing to measure customer impact
• End-to-end quality goes beyond shipping software
• It is about outcome, value for the customer
• How do you measure whether you solved a real customer need?

• Start at the story level, instead of WHAT you build you talk about
what customer need you solve and WHY that matters
• Come up with a Hypothesis and what you will measure to make it
quantative, meaningful?
 E-commerce industry is a great example

By VWO Blog

Optimizey Blog

By VWO Blog

Some of the A/B Testing tools
• Adobe Test & Target
• Optimizely
• Maxymizely
• and more https://conversionsciences.com/blog/ab-testing-tools
• or home-grown

The 8 rules of A/B Testing by Search Engine Watch
• Hypothesis
• One Variable at a time
• Clear and aligned success criteria
• Volume and statistical significance (also risk)
https://www.optimizely.com/optimization-glossary/statistical-significance
• Test Groups and Splits (can be 50/50, 10/90, etc.)
• Randomization
• Always be testing, with common sense
• Documentation

Q&A
http://www.topproducer.com/careers
12

Agenda
• Q&A – 5-6 min

Where in the stack to invest in test automation
• Always go to the lowest levels
• Always write small, reusable, fast tests

Experience
behavior
Functional
behavior
Data
behavior

Services

Test automation best practices
• Test Early and Test often
 Have strong acceptance criteria – is an outline for your testing
 Write your unit testing as components are created and iterate – full
or lightweight TDD
 Integrate the tests early on into your CI/CD so they run all the time
 Run them all the time!

• Create good quality test data
 Managing test data, especially at scale, is a hard problem to
manage
 Many times there is a pre-set environment with pre-set data which
lives for long-times and as time passes gets messy and in the way
for teams to be efficient

• Create good quality test data
 The nirvana is that test data gets generated as tests run, and
cleaned up afterwards
 And that you have sets of test data you can load into an
environment as it gets stood up
 And that you can spin up and tear down environments as needed
– really hard to achieve

• Create UI tests which are resistant to change
 Use names and class IDs to identify objects in the UI
 Never use text labels, they always change
 Never use positions, they always change

• Decide what test cases to automate, like
 Frequently used (key) features
 Tests which tend to cause human errors
 Repetitive tests run with each CI/CD build / deployment
 Tests you need to run on multiple platforms / browsers
 Tests which are very time consuming

• Review your test cases with PO / PM and Engineers
 Get many eyes on it and understand the main flows and edge
cases
 Iterate on the testing like you iterate on the development
 Get it out to customers for validation and use the feedback

• BDD – Behavior Driven Dev
 Test the behavior, business use case
• TDD – Test Driven Dev
 Be pragmatic, define the test for a component and then write the
code, not the whole thing
https://codeutopia.net/blog/2015/03/01/unit-testing-tdd-and-bdd

• What are other doing or not doing?

Q&A
26

Agenda
• Q&A – 5-6 min

Testing for reliability, resilience and recovery
• Reliability is a concern owned by your Services & APIs
 Clustering, fail-over
 Elastic scaling (up and down)
 Error handling
 More

Basic steps for reliability testing…
• Set up a test environment where ideally all services are set up for
elastic scaling (cloud)
• Run a set of automated tests to establish a baseline (can be really
any automated tests like Selenium or JMeter)
• Scale up and down the load and observe how the various services
will start to scale up and down
• It will also tell you which services are the most stressed parts of
your application

Basic steps for reliability testing – expected results
• You want to see that your various services scale up – if they don’t
then they will break (unless they are super-fast/efficient)
• You want to see that the services scale down again – if not then
your costs will go up and up
• During the scale up and down you want to see that the functionality
doesn’t break (validates things like state-less)

More reliability testing…
• You can also set the maximum for the scaling up and down and
observe how the system behaves as it reaches the maximum load
• It will point out the weaknesses in your architecture and where to
optimize

• Resilience is a concern owned by the consumer / client
 Error handling
 Graceful degradation of features, so if one or more services are
down the user can still use your application:
For example: If the Ratings & Review service is down I can still
find and view products, or purchase products, without delays

By Nextflix Blog

• The “golden standard” is Hystrix, former circuit-breaker, by Netflix
 https://github.com/Netflix/Hystrix/wiki
• Calls to services are wrapped in commands, and you can define
thresholds to these commands

By Nextflix Blog

Basics of resilience testing…
• Run automated tests, like Selenium or JMeter, and while these run
start shutting off non-essential services
• Then observe that your experience still functions, fast and reliable
• And observe that some of the features depending on these services
degrade – can be either not showing up or showing relevant
messages to users

• Recovery testing is the exercise of what it takes to recover from a
major fault or an outage
 Troubleshooting – like Logs, instrumentation, etc.
 Restoring services
 Worst case, restoring data
 Root causing is part of it

Basics of recovery testing…
• Run automated tests, like Selenium or JMeter, and while these run
start shutting off essential services
• Then observe that your experience stops functioning or slows down
for users
• Then run through the manual and automated steps to recover,
without making the situation worse for users

Q&A
41

Agenda
• Q&A – 5-6 min

Q&A
43

Reaching for Your Quality Stretch Goals: Testing at Realtor.com

More Related Content

What's hot

Similar to Reaching for Your Quality Stretch Goals: Testing at Realtor.com

Recently uploaded

Reaching for Your Quality Stretch Goals: Testing at Realtor.com