EdgeCast
Performance Engineering Group

Third Party Monitoring
Use Cases

Nathan Moore, Web Performance Engineer
nmoore@ed...
Web Performance Group

Analyze performance trends and come up with
ideas to make the CDN faster

© 2013 EDGECAST NETWORKS
...
Performance Optimizations
•
•
•
•

Network Stack
Routing
Kernel
Application

© 2013 EDGECAST NETWORKS

TCP Optimizations

...
Performance Optimizations
• We monitor as much as we can to find these:

Needles in haystacks

© 2013 EDGECAST NETWORKS

2...
Why monitor?

You can’t manage what you don’t measure

-- Deming
-- Drucker
-- Traditional
© 2013 EDGECAST NETWORKS

2850 ...
What can you monitor with Third Party
Applications?
• Performance
– Latency
– Bandwidth

• Availability
– Uptime

• Rate o...
Why care?

Web performance drives KPIs

© 2013 EDGECAST NETWORKS

2850 OCEAN PARK BLVD, SUITE 110, SANTA
MONICA CA

7
Why care?

KPIs:
• Fast websites drive Sales and Engagement
• Reliable websites drive Sales and Engagement

© 2013 EDGECAS...
Not just a made-up statistic

http://sites.google.com/site/glinden/Home/StanfordDataMining.2006-11-28.ppt?attredirects=0
©...
Why monitor?

What you can measure, you can optimize
-- Ilya Grigorik

© 2013 EDGECAST NETWORKS

2850 OCEAN PARK BLVD, SUI...
Simplified Internet Traffic Flow

© 2013 EDGECAST NETWORKS

2850 OCEAN PARK BLVD, SUITE 110, SANTA
MONICA CA

11
Categorizing the World

© 2013 EDGECAST NETWORKS

2850 OCEAN PARK BLVD, SUITE 110, SANTA
MONICA CA

12
Monitoring the World

© 2013 EDGECAST NETWORKS

2850 OCEAN PARK BLVD, SUITE 110, SANTA
MONICA CA

13
What are Third Party Monitors for

Synthetic monitoring is
needed to dynamically
monitor the External
Internet
© 2013 EDGE...
Monitoring Hooks for Synthetic Monitoring

© 2013 EDGECAST NETWORKS

2850 OCEAN PARK BLVD, SUITE 110, SANTA
MONICA CA

15
Synthetic Monitoring
Synthetic monitoring is clean
•
•
•
•

Small number of nodes
Well defined network
Highly reliable lin...
State of the Internet

© 2013 EDGECAST NETWORKS

2850 OCEAN PARK BLVD, SUITE 110, SANTA
MONICA CA

17
What are Third Party Monitors For

Real User Monitoring
(RUM) is needed to
measure what your end
users really see
© 2013 E...
From Synthetic to Real User Metrics (RUM)

Reporting
Server

CDN 1

© 2013 EDGECAST NETWORKS

CDN 2

2850 OCEAN PARK BLVD,...
Real User Data / RUM

© 2013 EDGECAST NETWORKS

2850 OCEAN PARK BLVD, SUITE 110, SANTA
MONICA CA

20
Real User Metrics (RUM)
RUM is messy. Rum is the real observed data generated
by real users on the real Internet on real w...
Real User Data / RUM benefits
Data aggregated across country and displayed across world

© 2013 EDGECAST NETWORKS

2850 OC...
Use Case 1:
What if you only have synthetic monitors?
New
platform
launch:
EdgeCast
Transact

© 2013 EDGECAST NETWORKS

28...
Use Case 1:
Launch of Transact
Directive from CEO:
Performance must match the existing matured network!

© 2013 EDGECAST N...
Use Case 1:
Launch of Transact
Transact is a brand new product from EdgeCast, designed to
accelerate e-commerce performanc...
Use Case 1:
Launch of Transact
Established monitors across the major third parties to observe
actual latencies worldwide

...
Use Case 1:
Launch of Transact
Compared performance between Transact and EdgeCast ADN at all nodes
Used the delta to deter...
Use Case 1:
Launch of Transact
Iterated: As issues were resolved, updated with new monitoring data
Repeat process!

© 2013...
Use Case 1:
Launch of Transact
Result: Worldwide average performance within 1.5ms
EdgeCast ADN

© 2013 EDGECAST NETWORKS

...
What are Third Party Monitors For

Third party monitoring
allows A/B comparison

© 2013 EDGECAST NETWORKS

2850 OCEAN PARK...
Use Case 2: RIPE/ATLAS
Using RIPE/ATLAS to evaluate the network for IPv6 launch
for World IPv6 Day, 2012

© 2013 EDGECAST ...
Use Case 2: RIPE/ATLAS
For World IPv6 Day, we deployed IPv6 across our entire
network.
The Challenge:
How does the IPv6 In...
Use Case 2: RIPE/ATLAS
http://atlas.ripe.net/about
RIPE Atlas is a global network of probes that measure Internet connecti...
Use Case 2: RIPE/ATLAS
JSON based API

© 2013 EDGECAST NETWORKS

Ability to run multiple tests

2850 OCEAN PARK BLVD, SUIT...
Use Case 2: RIPE/ATLAS
We launched a beacon dedicated to IPv6 measurements
which tested:
• IPv4 only
• IPv6 only
• Dual IP...
Use Case 2: RIPE/ATLAS

What did we learn from Atlas about our
IPv6 performance and availability?

© 2013 EDGECAST NETWORK...
Use Case 2: RIPE/ATLAS
IPv6 Reach Analysis

AS path analysis
© 2013 EDGECAST NETWORKS

AS path of failed traces
2850 OCEAN...
Use Case 2: RIPE/ATLAS
Visualization of Country-of-Origin traffic to EdgeCast POP

Ex: Ukraine is served by both POP7 and ...
Use Case 2: RIPE/ATLAS
Availability predictions before launch

© 2013 EDGECAST NETWORKS

2850 OCEAN PARK BLVD, SUITE 110, ...
Use Case 2: RIPE/ATLAS
Performance improvements during pre-launch
Conclusions: IPv6 still has room for improvement

© 2013...
Use Case 3: Analytics within Third Party Monitors
Using Third Party Monitors to answer real-world questions
Often, we’re a...
Use Case 3: Analytics within Third Party Monitors
Sales wanted to know why a prospective client was performing really well...
Use Case 3: Analytics within Third Party Monitors
There was a lot of variance within countries
EdgeCast

© 2013 EDGECAST N...
Use Case 3: Analytics within Third Party Monitors
The scatterplot shows some odd signs of horizontal banding.
EdgeCast

© ...
Use Case 3: Analytics within Third Party Monitors
Good clue: Looking at the total downloaded bytes instead of response tim...
Use Case 3: Analytics within Third Party Monitors
A histogram shows the magnitude of this difference

© 2013 EDGECAST NETW...
Use Case 3: Analytics within Third Party Monitors
Conclusion:
Client was returning different data depending on the geoIP l...
Questions?

© 2013 EDGECAST NETWORKS

2850 OCEAN PARK BLVD, SUITE 110, SANTA
MONICA CA

48
Upcoming SlideShare
Loading in...5
×

Synthetic and Real World Web Performance Monitoring

6,499

Published on

By Nathan Moore, EdgeCast/Verizon Web Performance Team

EdgeCast competes heavily on performance. We work very hard to make the Internet faster, more reliable, and more robust. One of the ways we do this is through our partnerships with third party monitors.

We measure our real-world performance constantly across the globe to ensure that we identify and react to any problems quickly and efficiently. The goal: eliminate any potential difficulty that can get between end users and their content.

Our friend and ally: third party monitors:

Our third party monitors help us to improve ourselves while providing the assurance and validation that our service legitimately performs to both our and our customer's expectations.

The below presentation was given recently at a small web performance conference, showing both the general operating theory behind our use of third party monitors and some specific use case examples demonstrating the value that they bring.

The use of synthetic and real world monitoring:

We use both synthetic and real world monitoring to ensure delivery quality and performance. This presentation shows three recent EdgeCast examples of how we deploy these testing techniques to optimize our vast, global network.

Transact: Launching a whole new global network

Launching a new product is always difficult. Doubly so when the word "worldwide" is attached to it. The challenge before us was to take our brand new Transact platform, complete with a brand new network and servers in over 20 brand new POPs around the world, and make it perform as well as our existing highly optimized network.

We relied extensively on third party monitors to both make this happen, and to demonstrate that we had succeeded.

IPv6: Deploying the future:

World IPv6 Day was designed to raise awareness that the future of the Internet is here, and that it's time to begin the needed transition away from IPv4 into the world of IPv6. EdgeCast participated, and deployed IPv6 throughout its extensive worldwide network.

But we needed to test: how did we know that we could deliver content across the IPv6 world? Using the RIPE/ATLAS probes to run worldwide traces gave us the data we needed to demonstrate that we could.

Competitive A/B testing: proving our performance:

It's not just the technology side of the business that needs to know how the EdgeCast network is performing. Our sales team depends heavily on the independent measurements provided by third party monitoring services.

This allows for A/B comparisons and independent confirmation of our performance claims. However, sometimes the monitoring services uncover some very odd and strange behavior, which requires some level of analysis to get at the root of.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,499
On Slideshare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
18
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Synthetic and Real World Web Performance Monitoring "

  1. 1. EdgeCast Performance Engineering Group Third Party Monitoring Use Cases Nathan Moore, Web Performance Engineer nmoore@edgecast.com
  2. 2. Web Performance Group Analyze performance trends and come up with ideas to make the CDN faster © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 2
  3. 3. Performance Optimizations • • • • Network Stack Routing Kernel Application © 2013 EDGECAST NETWORKS TCP Optimizations 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 3
  4. 4. Performance Optimizations • We monitor as much as we can to find these: Needles in haystacks © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 4
  5. 5. Why monitor? You can’t manage what you don’t measure -- Deming -- Drucker -- Traditional © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 5
  6. 6. What can you monitor with Third Party Applications? • Performance – Latency – Bandwidth • Availability – Uptime • Rate of Change of Internet – Snapshot of Internet as a whole © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 6
  7. 7. Why care? Web performance drives KPIs © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 7
  8. 8. Why care? KPIs: • Fast websites drive Sales and Engagement • Reliable websites drive Sales and Engagement © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 8
  9. 9. Not just a made-up statistic http://sites.google.com/site/glinden/Home/StanfordDataMining.2006-11-28.ppt?attredirects=0 © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 9
  10. 10. Why monitor? What you can measure, you can optimize -- Ilya Grigorik © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 10
  11. 11. Simplified Internet Traffic Flow © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 11
  12. 12. Categorizing the World © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 12
  13. 13. Monitoring the World © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 13
  14. 14. What are Third Party Monitors for Synthetic monitoring is needed to dynamically monitor the External Internet © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 14
  15. 15. Monitoring Hooks for Synthetic Monitoring © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 15
  16. 16. Synthetic Monitoring Synthetic monitoring is clean • • • • Small number of nodes Well defined network Highly reliable links between nodes and networks Shows health of the backbone of the Internet © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 16
  17. 17. State of the Internet © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 17
  18. 18. What are Third Party Monitors For Real User Monitoring (RUM) is needed to measure what your end users really see © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 18
  19. 19. From Synthetic to Real User Metrics (RUM) Reporting Server CDN 1 © 2013 EDGECAST NETWORKS CDN 2 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA CDN 3 19
  20. 20. Real User Data / RUM © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 20
  21. 21. Real User Metrics (RUM) RUM is messy. Rum is the real observed data generated by real users on the real Internet on real web browsers Combinatorics: • Hundreds of Millions of TCP connections • Tens of Millions of end users • Thousands of ISPs • Tens of backbone providers • One of your web servers • Data Distribution issues: • Power law distributions => averages lose explanatory power • Quantiles and Cumulative Density Functions (CDF) gain importance • Aggregating data and mining for correlations becomes more difficult © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 21
  22. 22. Real User Data / RUM benefits Data aggregated across country and displayed across world © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 22
  23. 23. Use Case 1: What if you only have synthetic monitors? New platform launch: EdgeCast Transact © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 23
  24. 24. Use Case 1: Launch of Transact Directive from CEO: Performance must match the existing matured network! © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 24
  25. 25. Use Case 1: Launch of Transact Transact is a brand new product from EdgeCast, designed to accelerate e-commerce performance on the web. Featuring full PCI compliance, it has its own dedicated and highly reliable network, its own dedicated servers, and a fully integrated security layer. The challenge is to deploy a brand new network, and make it perform like an existing, fully matured network. Third party monitors are the key to accomplish this, and to know when we’ve accomplished it. © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 25
  26. 26. Use Case 1: Launch of Transact Established monitors across the major third parties to observe actual latencies worldwide © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 26
  27. 27. Use Case 1: Launch of Transact Compared performance between Transact and EdgeCast ADN at all nodes Used the delta to determine where issues were © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 27
  28. 28. Use Case 1: Launch of Transact Iterated: As issues were resolved, updated with new monitoring data Repeat process! © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 28
  29. 29. Use Case 1: Launch of Transact Result: Worldwide average performance within 1.5ms EdgeCast ADN © 2013 EDGECAST NETWORKS Transact 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 29
  30. 30. What are Third Party Monitors For Third party monitoring allows A/B comparison © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 30
  31. 31. Use Case 2: RIPE/ATLAS Using RIPE/ATLAS to evaluate the network for IPv6 launch for World IPv6 Day, 2012 © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 31
  32. 32. Use Case 2: RIPE/ATLAS For World IPv6 Day, we deployed IPv6 across our entire network. The Challenge: How does the IPv6 Internet perform? We used RUM data provided by RIPE/ATLAS to find out. © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 32
  33. 33. Use Case 2: RIPE/ATLAS http://atlas.ripe.net/about RIPE Atlas is a global network of probes that measure Internet connectivity and reachability, providing an unprecedented understanding of the state of the Internet in real time. http://www.ripe.net The RIPE NCC is one of five Regional Internet Registries (RIRs) providing Internet resource allocations, registration services and coordination activities that support the operation of the Internet globally. © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 33
  34. 34. Use Case 2: RIPE/ATLAS JSON based API © 2013 EDGECAST NETWORKS Ability to run multiple tests 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 34
  35. 35. Use Case 2: RIPE/ATLAS We launched a beacon dedicated to IPv6 measurements which tested: • IPv4 only • IPv6 only • Dual IPv4 & IPv6 To reduce failures, the beacon also checked connectivity to ipv6.google.com © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 35
  36. 36. Use Case 2: RIPE/ATLAS What did we learn from Atlas about our IPv6 performance and availability? © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 36
  37. 37. Use Case 2: RIPE/ATLAS IPv6 Reach Analysis AS path analysis © 2013 EDGECAST NETWORKS AS path of failed traces 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 37
  38. 38. Use Case 2: RIPE/ATLAS Visualization of Country-of-Origin traffic to EdgeCast POP Ex: Ukraine is served by both POP7 and POP1, and POP7 has higher latency © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 38
  39. 39. Use Case 2: RIPE/ATLAS Availability predictions before launch © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 39
  40. 40. Use Case 2: RIPE/ATLAS Performance improvements during pre-launch Conclusions: IPv6 still has room for improvement © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 40
  41. 41. Use Case 3: Analytics within Third Party Monitors Using Third Party Monitors to answer real-world questions Often, we’re asked questions by different groups of the company. Sales, for instance, always wants to prove that we’re the fastest worldwide. But what to do when some regions are faster than others? © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 41
  42. 42. Use Case 3: Analytics within Third Party Monitors Sales wanted to know why a prospective client was performing really well in Asia, but not nearly as well in Europe. EdgeCast Asia Mystery Competitor X © 2013 EDGECAST NETWORKS EdgeCast 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA Europe Mystery Competitor X 42
  43. 43. Use Case 3: Analytics within Third Party Monitors There was a lot of variance within countries EdgeCast © 2013 EDGECAST NETWORKS Mystery Competitor X 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 43
  44. 44. Use Case 3: Analytics within Third Party Monitors The scatterplot shows some odd signs of horizontal banding. EdgeCast © 2013 EDGECAST NETWORKS Mystery Competitor X 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 44
  45. 45. Use Case 3: Analytics within Third Party Monitors Good clue: Looking at the total downloaded bytes instead of response time, the banding becomes really obvious, and wildly different in size EdgeCast © 2013 EDGECAST NETWORKS Mystery Competitor X 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 45
  46. 46. Use Case 3: Analytics within Third Party Monitors A histogram shows the magnitude of this difference © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 46
  47. 47. Use Case 3: Analytics within Third Party Monitors Conclusion: Client was returning different data depending on the geoIP lookup of the end user. So end users in different countries were deliberately being returned different objects. However, as caches, both EdgeCast and Mystery Competitor X were caching the index.html page, so whatever was on that page was cached and returned on every request. New data was only returned once every day, when the caching rule expired, and a fresh request to origin was made. The extreme difference in size was largely a function of random chance- we got lucky to return smaller objects during this sample time. © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 47
  48. 48. Questions? © 2013 EDGECAST NETWORKS 2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA 48
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×