Enterprise websites split across many regions and locales are huge, but every page matters. Keeping track of the website’s quality, across all pages and on different devices, is a complicated and time-consuming process. It requires the constant evaluation of page performance, web analytics implementation accuracy, accessibility, and SEO audits. Baresquare developed an automated solution to keep track of website changes and website quality. This is a scalable solution built on GCP, with cost and resources customized to the website's size. The solution utilizes a customized version of Lighthouse SDK and a BigQuery database for keeping track of results. Analyzing results is a breeze by connecting BigQuery to a dashboard or feeding data to other services. The talk will focus on the business needs addressed by the solution offered by Baresquare. How GCP, and the resources available, allowed the creation of this project. Freedom offered by the cloud and how it empowers data analysts to address business needs. https://tzamtzis.gr/2020/web-analytics/scalable-website-audit-for-analysts-using-the-cloud-dev-fest-hellas-2020/

1. Scalable website
audit for analysts
using the cloud
Panagiotis Tzamtzis
Head of Data-ops, Baresquare

2. Who is Baresquare?
Products & services focused on:
● Data analytics
● Action detection
● Smart workflows
● Action triggering
“Make Data Worthwhile by Focusing on Action-Driven Analytics”

3. Goals of this project
1. Crawl large websites automatically
2. Create a scalable solution
3. CI/CD and cooperation of multiple team members
4. Secure and scalable data storage
5. Fast and easy data analysis
6. Short development time

4. Difficulties of the project
1. Corporate websites are huge and span across several countries
2. Frequent audits and historical data are necessary
3. There’s a long list of things which can go wrong
4. Standards on the web change very often and their scope is broad
5. Fixing an issue can be a complicated process
6. On the web it does matter what your competitors are doing

5. Objective:
Use a smart automated approach to
scale down the problem into a data
analytics use case

6. Let’s get technical
Infrastructure
● Scalable
infrastructure
● Secure &
compatible tools
● Consistent
availability
Testing framework
● Automated tests
● Quick onboarding
● Future-proof
● Rich insights
Web browser
● Browser API
● Emulate
browsers/screens
● Act as a normal
browser

7. Solution diagram
1.Prepare
audit
Start
Create
VM
Find
URLs
Store
URLs
Publish
URLs
Start
workers
Turn off
this VM
2.Complete
audit
Store
reports
Lighthouse
+ Puppeteer
Lighthouse
+ Puppeteer
Get
URLs
Store
data
Turn off
All VMs

8. Reporting on the results
1. Lighthouse reports
2. Data studio dashboard
3. External analysis

9. Lessons learned
1. Compute engine instance is ideal for parallelizing long running tasks
2. Use “Instance metadata” to keep track of execution
3. BigQuery partitioning is important for cost
4. Data studio + BigQuery = 💗, even in real-time
5. Try using cloud tools, instead of re-inventing the wheel
6. Cloud tools have artificial limits, don’t perceive them as blockers
7. Lighthouse is more than a predefined set of SEO & performance audits
8. Take some time to study the details of each cloud tool

10. The Team
Michalis Samaras,
Automations engineer
- Product integrations
- Business requirements
Panagiotis Tzamtzis,
Head of Data-ops
- Prototype implementation
- Data architecture
Lucas Zamboulis,
Lead Cloud Architect
- Product development
- Cloud architecture

11. Useful links
● Baresquare
● Google Lighthouse
● Puppeteer
● Panagiotis’ personal website
● Panagiotis LinkedIn
● Panagiotis SlideShare