Synthetic and Real World Web Performance Monitoring

EdgeCast
Performance Engineering Group

Third Party Monitoring
Use Cases

Nathan Moore, Web Performance Engineer
nmoore@edgecast.com

Web Performance Group

Analyze performance trends and come up with
ideas to make the CDN faster

© 2013 EDGECAST NETWORKS

2850 OCEAN PARK BLVD, SUITE 110, SANTA
MONICA CA

2

Performance Optimizations
•
•
•
•

Network Stack
Routing
Kernel
Application


TCP Optimizations

MONICA CA

3

Performance Optimizations
• We monitor as much as we can to find these:

Needles in haystacks


MONICA CA

4

Why monitor?

You can’t manage what you don’t measure

-- Deming
-- Drucker
-- Traditional

MONICA CA

5

What can you monitor with Third Party
Applications?
• Performance
– Latency
– Bandwidth

• Availability
– Uptime

• Rate of Change of Internet
– Snapshot of Internet as a whole

MONICA CA

6

Why care?

Web performance drives KPIs


MONICA CA

7

Why care?

KPIs:
• Fast websites drive Sales and Engagement
• Reliable websites drive Sales and Engagement


MONICA CA

8

Not just a made-up statistic

http://sites.google.com/site/glinden/Home/StanfordDataMining.2006-11-28.ppt?attredirects=0

MONICA CA

9

Why monitor?

What you can measure, you can optimize
-- Ilya Grigorik


MONICA CA

10

Simplified Internet Traffic Flow


MONICA CA

11

Categorizing the World


MONICA CA

12

Monitoring the World


MONICA CA

13

What are Third Party Monitors for

Synthetic monitoring is
needed to dynamically
monitor the External
Internet

MONICA CA

14

Monitoring Hooks for Synthetic Monitoring


MONICA CA

15

Synthetic Monitoring
Synthetic monitoring is clean
•
•
•
•

Small number of nodes
Well defined network
Highly reliable links between nodes and networks
Shows health of the backbone of the Internet


MONICA CA

16

State of the Internet


MONICA CA

17

What are Third Party Monitors For

Real User Monitoring
(RUM) is needed to
measure what your end
users really see

MONICA CA

18

From Synthetic to Real User Metrics (RUM)

Reporting
Server

CDN 1


CDN 2

MONICA CA

CDN 3

19

Real User Data / RUM


MONICA CA

20

Real User Metrics (RUM)
RUM is messy. Rum is the real observed data generated
by real users on the real Internet on real web browsers
Combinatorics:
• Hundreds of Millions of TCP connections
• Tens of Millions of end users
• Thousands of ISPs
• Tens of backbone providers
• One of your web servers
• Data Distribution issues:
• Power law distributions => averages lose explanatory power
• Quantiles and Cumulative Density Functions (CDF) gain importance
• Aggregating data and mining for correlations becomes more difficult


MONICA CA

21

Real User Data / RUM benefits
Data aggregated across country and displayed across world


MONICA CA

22

Use Case 1:
What if you only have synthetic monitors?
New
platform
launch:
EdgeCast
Transact


MONICA CA

23

Use Case 1:
Launch of Transact
Directive from CEO:
Performance must match the existing matured network!


MONICA CA

24

Use Case 1:
Launch of Transact
Transact is a brand new product from EdgeCast, designed to
accelerate e-commerce performance on the web.
Featuring full PCI compliance, it has its own dedicated and highly
reliable network, its own dedicated servers, and a fully
integrated security layer.
The challenge is to deploy a brand new network, and make it
perform like an existing, fully matured network.
Third party monitors are the key to accomplish this, and to know
when we’ve accomplished it.

MONICA CA

25

Use Case 1:
Launch of Transact
Established monitors across the major third parties to observe
actual latencies worldwide


MONICA CA

26

Use Case 1:
Launch of Transact
Compared performance between Transact and EdgeCast ADN at all nodes
Used the delta to determine where issues were


MONICA CA

27

Use Case 1:
Launch of Transact
Iterated: As issues were resolved, updated with new monitoring data
Repeat process!


MONICA CA

28

Use Case 1:
Launch of Transact
Result: Worldwide average performance within 1.5ms
EdgeCast ADN


Transact

MONICA CA

29

What are Third Party Monitors For

Third party monitoring
allows A/B comparison


MONICA CA

30

Use Case 2: RIPE/ATLAS
Using RIPE/ATLAS to evaluate the network for IPv6 launch
for World IPv6 Day, 2012


MONICA CA

31

For World IPv6 Day, we deployed IPv6 across our entire
network.
The Challenge:
How does the IPv6 Internet perform? We used RUM data
provided by RIPE/ATLAS to find out.


MONICA CA

32

http://atlas.ripe.net/about
RIPE Atlas is a global network of probes that measure Internet connectivity and
reachability, providing an unprecedented understanding of the state of the Internet in
real time.

http://www.ripe.net
The RIPE NCC is one of five Regional Internet Registries (RIRs) providing Internet
resource allocations, registration services and coordination activities that support
the operation of the Internet globally.

MONICA CA

33

JSON based API


Ability to run multiple tests

MONICA CA

34

We launched a beacon dedicated to IPv6 measurements
which tested:
• IPv4 only
• IPv6 only
• Dual IPv4 & IPv6

To reduce failures, the beacon also checked connectivity
to ipv6.google.com


MONICA CA

35


What did we learn from Atlas about our
IPv6 performance and availability?


MONICA CA

36

IPv6 Reach Analysis

AS path analysis

AS path of failed traces
MONICA CA

37

Visualization of Country-of-Origin traffic to EdgeCast POP

Ex: Ukraine is served by both POP7 and POP1, and POP7 has higher latency

MONICA CA

38

Availability predictions before launch


MONICA CA

39

Performance improvements during pre-launch
Conclusions: IPv6 still has room for improvement


MONICA CA

40

Use Case 3: Analytics within Third Party Monitors
Using Third Party Monitors to answer real-world questions
Often, we’re asked questions by different groups of the company.
Sales, for instance, always wants to prove that we’re the fastest worldwide.
But what to do when some regions are faster than others?


MONICA CA

41

Sales wanted to know why a prospective client was performing really well
in Asia, but not nearly as well in Europe.

EdgeCast

Asia
Mystery Competitor X


EdgeCast

MONICA CA

Europe

42

There was a lot of variance within countries
EdgeCast



MONICA CA

43

The scatterplot shows some odd signs of horizontal banding.
EdgeCast



MONICA CA

44

Good clue: Looking at the total downloaded bytes instead of response time,
the banding becomes really obvious, and wildly different in size

EdgeCast



MONICA CA

45

A histogram shows the magnitude of this difference


MONICA CA

46

Conclusion:
Client was returning different data depending on the geoIP lookup
of the end user. So end users in different countries were
deliberately being returned different objects.
However, as caches, both EdgeCast and Mystery Competitor X were
caching the index.html page, so whatever was on that page was
cached and returned on every request. New data was only returned
once every day, when the caching rule expired, and a fresh request
to origin was made.
The extreme difference in size was largely a function of random
chance- we got lucky to return smaller objects during this sample
time.


MONICA CA

47

Questions?


MONICA CA

48

Synthetic and Real World Web Performance Monitoring

Recommended

Recommended

More Related Content

Similar to Synthetic and Real World Web Performance Monitoring

Similar to Synthetic and Real World Web Performance Monitoring (20)

Recently uploaded

Recently uploaded (20)

Synthetic and Real World Web Performance Monitoring