The document details the implementation of automated visual testing for the NSW Government website to ensure visual consistency across browsers and devices after each release. It discusses utilizing artificial intelligence and automation tools to enhance the efficiency and coverage of visual testing, moving away from labor-intensive manual processes. The automated workflow includes synchronization of production content, baseline image capture, and testing in various environments with a structured deployment process.

1. Automated Visual
Testing in nsw.gov.au
Sabbir Subhan

2. Digital Channels Snapshot – October 2021
Data source: Google Analytics nsw.gov.au & NSW Government Facebook
Monthly averages October 2021
8.8M
Average visitors per
month
58.8% new visitors
18.4M
Average visits
per month
2m 11s average session duration
38.8M
Average total page views
per month
2.09 pages/session

3. Problem Statement
3
We need ensure that nsw.gov.au looks visually perfect across supported browsers and mobile
devices after each new release into Production.
Insert > Header & Footer > tick Footer box > Apply to all |

4. What is Visual Testing
4
Visual testing is the manual process of comparing a website under test against a baseline
enviroment (usually Production system).
Manual visual testing is resource intensive and coverage is often low due QA resource
constraint. Again, manual visual testing prone to subjectivity of QA resource and very
repetitive tasks.
Modern approaches of visual testing have incorporated artificial intelligence and
automation, known as Visual AI, to view as a human eye would and avoid false positives.
Insert > Header & Footer > tick Footer box > Apply to all |

5. Solution Approach
5
We like to automate the manual visual testing so that we can scale up the visual testing
effort, increase the coverage and efficiency.
Key points:
1) We will leverage exiting Testing Infrastructure (AWS Automation Server) to run the
Automated Visual Testing.
2) We will use same automation framework (Java and Selenium) to automate browser
navigation and interactions.
3) We are proposing to use a commercial visual testing platform.
Insert > Header & Footer > tick Footer box > Apply to all |

6. Automated Visual Testing Workflow
6
1. Production content are synced to QA and Stage around 1 AM everyday.
2. Baseline images are taken at 8 AM Tuesday from PROD in every weekday.
3. Baseline Images are stored in Applitools platform.
4. Above process is 100% automated.
5. New version is deployed to QA when pull request is merged.
6. New release is deployed to Stage at 12 PM Tuesday.
7. QA team triggers visual test on QA and Stage (as required)
8. QA team triages visual differences identified by Applitools
Insert > Header & Footer > tick Footer box > Apply to all |

7. 7
Deployment Process
Preview
Deployment
Tasks:
• PR to Preview (DEV
Team)
• Apply label for
inclusion in release
(P.O)
Preview
Environment
Testing
Tasks:
• Functional/Cross
Browser Testing (QA
Team)
• Content Review
(Content Team)
• Design Review (UX
Team)
• User Acceptance
Testing (P.O)
• Code Review (Dev
Team)
• Refactor (Dev Team)
QA
Deployment
Tasks:
• PR Merge to QA (QA
Team)
QA
Environment
Testing
Tasks:
• Functional/Cross
Browser Testing (QA
Team)
• Refactor Refinement
(Dev Team)
If cant be tested in
M8s Environment
•Design Review (UX
Team)
•User Acceptance
Testing (P.O)
•Refactor (Dev Team)
Staging
Deployment
Tasks:
• PR Merge to Staging
(QA Team)
Staging
Environment
Testing and QA
Tasks:
• Functional/Cross
Browser Testing (QA
Team)
• Business Verification
Testing (QA Team)
• Final UAT (P.O)
• Change Meeting (CM
Group)
Production
Deployment
Tasks:
•PR Merge to
Production (Dev DCS)
•BVT (QA Team)
Code Freeze
Monday 12:00 pm
1 2 3 4 5 6 7
Staging merge
Tuesday 12:00 pm
Prod deployment
Tuesday 8:00 pm
Estimated Time:
40 Mins
Estimated Time:
3 hours
Estimated Time:
40 Mins
Estimated Time:
2 hours
Estimated Time:
40 Mins
Estimated Time:
90 Mins
Estimated Time:
60 Mins

8. Key data on Visual testing
8
1. Around 900 visual Tests.
2. Tests are executed on three view ports: Desktop, Mobile and Tablet.
3. Code is developed using Cucumber, Java and Selenium.
4. Advanced features such
• Ignore section of a page
• Dynamic visual AI logic such Layout mode
• Selenium interaction with get desired state of a page
• Visual Testing for a region of a page
Insert > Header & Footer > tick Footer box > Apply to all |

9. Code Snippet
9
Insert > Header & Footer > tick Footer box > Apply to all |

10. Layout Match Level
10
Insert > Header & Footer > tick Footer box > Apply to all |

11. Ignore Section
11
Insert > Header & Footer > tick Footer box > Apply to all |

12. Visual Testing on a Region of Page
12
Insert > Header & Footer > tick Footer box > Apply to all |

13. Some real live bug catching
13
Insert > Header & Footer > tick Footer box > Apply to all |