This session looks at common issues with just relying on end to end automation testing tools, using examples to demonstrate common pitfalls and how visual testing can help add another tool to your tool belt. The talk looks at why we automate tests, the issue with just manually testing, common end to end automation pitfalls, a brief introduction to visual testing and finally a look at common issues with visual testing and ways to overcome them. Through the use of interactive examples the audience will gain an understanding of why relying on just manual testing can become an issue and how too much automation has a negative impact by looking at testing anti-patterns. The audience will also learn what visual testing is, what tools are available, some of the common pitfalls of using visual testings as well as tips on ways to overcome them based on experience of creating a custom visual test framework at my current employer.

1. Spot the difference Automating visual
regression testing
@11vlr

2. @11vlr
Spot the difference Automating visual
regression testing

3. @11vlr
• Previously a Software Engineer for
7 years (.Net C#)
• Currently a Senior Test Engineer
@Vizolution
• Code Club volunteer
• Organiser of SwanseaCon
• Co-organiser of DDD Wales
ABOUT ME

4. • Automated testing is more reliable, as
it is performed by tools and/or scripts.
• Automated testing is executed by
software tools, so it is significantly
faster than a manual approach.
WHY AUTOMATE END
TO END UI TESTS?
@11vlr

5. @11vlr
• Investment is required
• Automated testing is a practical option
when the test cases are run
repeatedly over a long time period.
• Automated testing entail
human observation and cannot
guarantee user-friendliness or positive
customer experience.
COMMON UI
AUTOMATION ISSUES
DOES NOT

6. IT CAN BE A SLIPPERY SLOPE
@11vlr

7. ICE-CREAM CONE ANTI-PATTERN
@11vlr
Unit
Service
UI
Manual &
Exploratory
Testing
£
£££

8. HOUR GLASS ANTI-PATTERN
@11vlr
£
£££
£
£££
UI
Unit
Manual &
Exploratory
Testing

9. @11vlr
WHEN YOU HAVE
UNIT BUT NO
INTEGRATION TESTS

10. @11vlr
TESTINGFRAGILE

11. TESTING PYRAMID
@11vlr
£
£££
UI
Service
Unit
Manual &
Exploratory
Testing

12. @11vlr
WHY NOT JUST STICK
WITH MANUAL
TESTING THEN?
• Manual testing allows for human
observation, which may be more
useful if the goal is user-friendliness
or improved customer experience.
UI
Service
Unit
Manual &
Exploratory
Testing

13. @11vlr
• Can be time intensive
• Is not accurate at all times due to
human error, hence it is less reliable.
• Is time-consuming, taking up human
resources.
• Is only practical when the test cases
are run once or twice, and frequent
repetition is not required.
MANUAL TESTING

14. WE DON’T HAVE TIME TO AUTOMATE
@11vlr

15. CATCH 22
@11vlr
We don’t automate
We don’t have time
BECAUSE BECAUSE

16. @11vlr
MANY OPTIONS

17. @11vlr
• Open source; anybody can see the
code and help
• Big community of support
• Can operate on almost any OS
• Multiple language support (Python,
Ruby, .Net, PHP, Java, JavaScript)
• Supports a range of browsers like
Opera, Chrome, IE (6,7,8,9,10,11),
Mozilla Firefox, Safari
SELENIUM

18. IT’S NOT A SILVER BULLET
@11vlr

19. @11vlr
• Speed to execute tests
• Render differently
BROWSER
COMPATIBILITY
ISSUES

20. EXECUTION SPEED
@11vlr

21. MAINTAINABILITY
@11vlr

22. @11vlr
HOW VALUABLE ARE THESE TESTS?

23. @11vlr
WHAT THE….







24. @11vlr
ALL IS NOT WHAT IT SEEMS







25. @11vlr
MAKING TESTS LESS
FLAKY

26. @11vlr
• Page object patterns
MAKING TESTS LESS
FLAKY
http://martinfowler.com/bliki/PageObject.html

27. @11vlr
• Qa-selectors
MAKING TESTS LESS
FLAKY

28. @11vlr
Qa- selectors

29. @11vlr
HOW GOOD ARE YOU AT SPOT THE
DIFFERENCE

30. @11vlr
HOW MANY DIFFERENCES CAN YOU FIND?

31. @11vlr
HOW MANY DIFFERENCES DID YOU FIND?
@11vlr

32. THERE WERE 8 DIFFERENCES
@11vlr

33. HOW DO YOU KNOW WHEN YOU’VE
FOUND THEM ALL?
@11vlr

34. CAN WE ADD MORE VALUE TO OUR
TESTS?
@11vlr
UI
Service
Unit
Manual &
Exploratory
Testing

35. WHAT IF WE COULD SIMPLY
@11vlr
equalsAssert

36. VISUAL TESTING
@11vlr

37. @11vlr
• Didn’t work straight out of the box
and/or contained lots of bloat
• Steep learning curve
• Small community so lack of support
• Needed to work with our current
automation framework
• Costs
ISSUES WITH OTHER
OFFERINGS

38. @11vlr
• Still in development
• Facing many challenges which are
because of the products we offer, the
ways in which we have developed
them and the way in which we test
them
ROLLING OUR OWN

39. @11vlr
• Browser rendering issues
RELIABILITY

40. @11vlr
EXECUTION SPEED

41. @11vlr
HUMAN VS MACHINE

42. @11vlr
HUMAN VS MACHINE

43. @11vlr
LETS ZOOM IN A LITTLE

44. @11vlr
LETS ZOOM IN A LITTLE BIT MORE…

45. @11vlr
GOT HIM!

46. @11vlr
HOW QUICK WAS THE MACHINE?

47. @11vlr
MAINTAINABILITY
Old code

48. @11vlr
MAINTAINABILITY
Old code
New code using some helpers

49. @11vlr
MAINTAINABILITY
New code using Specflow
New code using some helpers

50. @11vlr
EASILY IDENTIFY THE DIFFERENCES

51. @11vlr
ACCURACY

52. @11vlr
IMAGE QUALITY DIFFERENCE

53. @11vlr
CHALLENGES SO FAR

54. @11vlr
MAINTAINING BASE
IMAGES
Keep a base image for each driver
type

55. @11vlr
SCREENSHOTS
• Viewport (default - WebDriver
W3C standard)
• Full Screen (stitching)
• Capture images of just
elements

56. @11vlr
SCREEN RESOLUTIONS

57. @11vlr
SCREEN RESOLUTIONS

58. @11vlr
Responsive design

59. @11vlr
HARDWARE ISSUES
Setting a tolerance for the 256
different intensities

60. @11vlr
STORAGE
Database / Online storage – file
sizes (many images)

61. @11vlr
DYNAMIC CONTENT

62. @11vlr
DYNAMIC CONTENT
class=“attendeesLink"

63. @11vlr
MOVING ELEMENTS
• CSS animations
• Videos

64. @11vlr
VISUALLY CHECKING DOCUMENTS

65. @11vlr
PDF DYNAMIC
CONTENT
• A way to blanket content within
documents

66. UPCOMING CHALLENGES
@11vlr

67. @11vlr
MORE COMPARISON
OPTIONS
• Text only
• Layout only

68. ANOTHER TOOL TO ADD TO YOUR BELT
@11vlr

69. @11vlr
QUESTIONS?
http://bit.ly/VisualTesting
@11VLR
WWW.VIVRICHARDS.CO.UK