Bots vs. Real Users – Page Load Times
You want/need to know about the Load Times and Core Web Vitals of
your Digital Brand or offerings for your real users?
Cool, let’s get there but be aware of the Bots. They can ruin
everything.
Best possible Source: Browser of your User
Best possible Analytics Engine =
Get Browser Performance Metrics into New
Relic
(Browser performance metrics arrived in New Relic)
1.) Create a free Account at https://newrelic.com/signup
2.) Follow the instructions on the start page – in case you are already a New Relic user – just click on
Be aware: This includes every User Agent
Make sense of the of the data
• Get an overview
Once you have collected at least a day of data you are good to start identifying your
real Real Users. Lets click on “Dashboards” create a new one and do some queries:
FROM PageView SELECT count(*) where domain = ' mainDomain.com ' SINCE 1 day AGO limit 20 timeseries
FROM PageView SELECT average(duration) where domain= mainDomain.com' SINCE 1 day AGO limit 20 timeseries
Make sense of the of the data : Find the real users
95% of Real Users are using up to date Browsers (more or less) – 80% of Bots (or
Machines) visit you with outdated Browser Versions – How to get there? Let’s group
them!
FROM PageView
select count(userAgentName) where domain
= ‘mainDomain.com' facet userAgentName
, userAgentVersion
limit 20
These are pretty
outdated
Versions
Make sense of the of the data : Proof for Bots
Bots behave different from Real People – they either appear in spikes (crawls) or
permanent (i. e. Synthetic Monitors). Let’s check how these Browserversions behave
FROM PageView
select count(userAgentName) where domain
= ‘mainDomain.com' facet userAgentName
, userAgentVersion
limit 20 since 1 day ago timeseries
Linear Traffic over the course of
the Day = not real users
Make sense of the of the data : Proof for Humans
Bots behave different from Real People – they either appear in spikes (crawls) or
permanent (i. e. Synthetic Monitors). Let’s check how these Browserversions behave
FROM PageView
select count(userAgentName) where domain
= ‘mainDomain.com' facet userAgentName
, userAgentVersion
limit 20 since 1 day ago timeseries
Normal Traffic – Human sleep at
Night. International variances
possible
Make sense of the data: Now the performance!
• We combine the learnings and check what’s going on. Bots often call from Data Centers while Real
People often struggle with bandwidth variances.
FROM PageView
SELECT
filter(average(duration), where userAgentVersion >= '83' and userAgentName like 'Chrome') as 'Chrome Real User‘,
filter(average(duration), where userAgentName not like 'Chrome' and userAgentVersion not like '13.2.3' and userAgentVersion not like '9.1' and userAgentVersion != '10') as 'Other Real User Brow
filter(average(duration), where domain = 'newrelic.com‘) as 'Duration Unfiltered‘,
filter(average(duration), where userAgentVersion < '83' and userAgentName like 'Chrome') as 'Bots‘
WHERE domain = ‘mainDomain.com‘ SINCE 7 days AGO timeseries 15 minutes
Here we have proof that:
• Bots are served way faster than humans
• The high amount of Bots (in this extreme
example) make the Performance appear solid
and stable
• It is helpful to understand what you are
looking for

Bots vs real_users

  • 2.
    Bots vs. RealUsers – Page Load Times You want/need to know about the Load Times and Core Web Vitals of your Digital Brand or offerings for your real users? Cool, let’s get there but be aware of the Bots. They can ruin everything.
  • 3.
    Best possible Source:Browser of your User Best possible Analytics Engine =
  • 4.
    Get Browser PerformanceMetrics into New Relic (Browser performance metrics arrived in New Relic) 1.) Create a free Account at https://newrelic.com/signup 2.) Follow the instructions on the start page – in case you are already a New Relic user – just click on
  • 5.
    Be aware: Thisincludes every User Agent
  • 6.
    Make sense ofthe of the data • Get an overview Once you have collected at least a day of data you are good to start identifying your real Real Users. Lets click on “Dashboards” create a new one and do some queries: FROM PageView SELECT count(*) where domain = ' mainDomain.com ' SINCE 1 day AGO limit 20 timeseries FROM PageView SELECT average(duration) where domain= mainDomain.com' SINCE 1 day AGO limit 20 timeseries
  • 7.
    Make sense ofthe of the data : Find the real users 95% of Real Users are using up to date Browsers (more or less) – 80% of Bots (or Machines) visit you with outdated Browser Versions – How to get there? Let’s group them! FROM PageView select count(userAgentName) where domain = ‘mainDomain.com' facet userAgentName , userAgentVersion limit 20 These are pretty outdated Versions
  • 8.
    Make sense ofthe of the data : Proof for Bots Bots behave different from Real People – they either appear in spikes (crawls) or permanent (i. e. Synthetic Monitors). Let’s check how these Browserversions behave FROM PageView select count(userAgentName) where domain = ‘mainDomain.com' facet userAgentName , userAgentVersion limit 20 since 1 day ago timeseries Linear Traffic over the course of the Day = not real users
  • 9.
    Make sense ofthe of the data : Proof for Humans Bots behave different from Real People – they either appear in spikes (crawls) or permanent (i. e. Synthetic Monitors). Let’s check how these Browserversions behave FROM PageView select count(userAgentName) where domain = ‘mainDomain.com' facet userAgentName , userAgentVersion limit 20 since 1 day ago timeseries Normal Traffic – Human sleep at Night. International variances possible
  • 10.
    Make sense ofthe data: Now the performance! • We combine the learnings and check what’s going on. Bots often call from Data Centers while Real People often struggle with bandwidth variances. FROM PageView SELECT filter(average(duration), where userAgentVersion >= '83' and userAgentName like 'Chrome') as 'Chrome Real User‘, filter(average(duration), where userAgentName not like 'Chrome' and userAgentVersion not like '13.2.3' and userAgentVersion not like '9.1' and userAgentVersion != '10') as 'Other Real User Brow filter(average(duration), where domain = 'newrelic.com‘) as 'Duration Unfiltered‘, filter(average(duration), where userAgentVersion < '83' and userAgentName like 'Chrome') as 'Bots‘ WHERE domain = ‘mainDomain.com‘ SINCE 7 days AGO timeseries 15 minutes Here we have proof that: • Bots are served way faster than humans • The high amount of Bots (in this extreme example) make the Performance appear solid and stable • It is helpful to understand what you are looking for