The document discusses lean approaches to UI/UX testing including rapid wireframing tools, guerrilla testing, and automated A/B testing to test ideas quickly and cheaply. It then introduces Applitools, a visual automation test tool that can automate visual testing to check for issues like anti-aliasing, brightness, and sub-pixel shifts across browsers, operating systems, and screen resolutions. The document outlines Applitools' workflow for baseline creation and test execution and provides a demo. It notes challenges with visual testing and how Applitools addresses them through specialized image processing.

1. Lean Quality &
Engineering
Customized for @Jitta - Mek Srunyu Stittri

2. Agenda
Lean UI/UX AB Testing
Visual Automation Test Tool (Applitools)
Lets talk about testing and scale

3. LEAN UI/UX
You don’t need a Masters
degree from Carnegie Mellon
to practice user experience.

4. Testing ideas quicker and cheaper
Engineers are expensive.
Designers are expensive.
How do you mitigate that cost?

5. We are not GOOG, FB or AAPL
Usability testing labs, eye tracking machines,
and big budgets to spend on bringing in
people that fit your target audience.
That’s. Not. Lean.

6. Testing ideas quicker and cheaper
◎ Rapid wireframing tools - Wireframe Phase
○ e.g. Balsamiq
◎ Guerilla Testing
◎ Data Driven A/B automation
○ Multi-version deployments in parallel
+ Use tools to measure user interaction
e.g. Mix Panel

7. Rapid Wireframing
https://balsamiq.com/
◎ Same feel of sketching on a whiteboard, but using a
computer.
◎ Iterate Sooner & Painlessly.
◎ Mockups requires little investment in time and effort,
revise your designs more often and refine them
sooner.
◎ More time for discussion and writing code.

8. Guerilla Testing
The Starbucks usability test: sit down in a
local coffee shop, set up your laptop, and offer
someone a free coffee to spend 10 minutes
using your product.
You would be surprised how many people are willing to
do it. Test lasts about 5-10 mins.

9. Guerilla Testing
Choosing Location/area is key. Which
demographics do you want.
“We did the test at The Creamery in San Francisco, and we
got a demographic that was a lot less techie than you
would think.”

10. Automated A/B testing

11. Automated A/B testing
All websites on the web have a goal - a reason for them to exist
◎ eCommerce websites want visitors buying products
◎ SaaS web apps want visitors signing up for a trial and converting to paid
visitors
◎ News and media websites want readers to click on ads or sign up for
paid subscriptions
Every business website wants visitors converting from just visitors to
something else. The rate at which a website is able to do this is its
"conversion rate". Measuring the performance of a variation (A or B) means
measuring the rate at which it converts visitors to goal achievers

12. Automated A/B testing
Why Should You do A/B Test?
A/B testing allows you to make more out of
your existing traffic. While the cost of
acquiring paid traffic can be huge, the cost of
increasing your conversions is minimal.

13. Automated A/B testing
1. Separate users into 2 controls groups.
2. Fork traffic to two different
implementations.
Did you remember the Microservice talk ? :)
3. Track and measure.
4. Measure, measure,
measure..
https://vwo.com/ab-testing/

14. Visual Automation testing
What is Visual Test?
A quality assurance activity aimed to verify
that a Graphical User Interface appears
correctly to users

15. A Visual Bug
This

16. A Visual Bug
Or this

17. Why automate visual testing?
THE TEST MATRIX IS TOO BIG TO
APPROXIMATE MANUALLY
◎ Web browsers
◎ Operating systems
◎ Screen resolutions
◎ Responsive designs
◎ L10n

18. Why automated visual testing?
Data Layer
Backend
Front end
Top of the stack checkpointing.
Requires the least amount of investment but huge
returns.

19. Why automated visual testing?

20. Why automated visual testing?
Many are already doing it (and sharing)…

21. Challenges

22. Challenges
If it's so important.
How come it's not
widely implemented ?
Its not that simple..

23. Anti Aliasing

24. Anti Aliasing

25. Brightness

26. Brightness

27. Sub pixel shifts

28. Sub pixel shifts

29. And more..
◎ Dynamic content
◎ 1 pixel offsets in element
positioning
◎ Moving elements
◎ Images of different size

30. Then why tell me to do it
There is a tool that can do all this for you..
Introducing - APPLITOOLS, A specialized image processing stack
designed to compare computer generated UI images
1. Anti-aliasing
2. Partial and full pixel offsets
3. Images of different size
4. Dynamic content
5. Moving elements
6. Layout matching

31. Workflow

32. Workflow
Testdata (BaseLine) Creation Flow
1. Identify a Panel and create a new Applitools UI testcase to navigate to the Panel under test,
do UI interactions and add Visual Checkpoints
2. Run the test once to generate the BaseLine
3. Go to Applitools to approve the BaseLine and add customizations as necessary
Test Execution Flow
1. Run the test
2. Analyze the test result on differences reported

33. Demo
https://eyes.applitools.com/app/tutorial.html

34. Testing at Scale (Next time)
◎ Manual Test (75% of startups fall here)
◎ Automated
○ Unit Test
○ Integration and functional Test
◉ API - Restful web service validation
◉ UI - Selenium and etc.

35. Resources
Lean UX
Laura Klein : UX for Lean Startups
Usability Counts : http://www.usabilitycounts.com/2013/07/17/ux-for-lean-startups-a-great-resource-for-founders-
and-new-designers/
VWO labs : https://vwo.com/features/
Automated Visual Testing that Doesn't Suck!
Adam Carmi - VP Engineering, Co-Founder at Applitools
http://www.meetup.com/SeleniumSanJose/events/209029032/ - November 17, 2014
http://www.slideshare.net/adamcarmi/visual-test-automation-that-does-not-suck - Adam’s original slide deck from
Selenium Meetup