Score card at Commsday Australia
•Download as PPTX, PDF•
1 like•326 views
At the Commsday event in Sydney April 4th 2016, Thomas Vasen, Subscriber Experience Evangelist at Procera Networks, made the case on how operators should change how they measure the quality their network is capable of delivering in order to fight back the OTT players’ Scorecards. And I’ll bring you’re a case study how this was done successfully.
Recommended
More Related Content
What's hot
What's hot (18)
Similar to Score card at Commsday Australia
Similar to Score card at Commsday Australia (20)
Recently uploaded
Recently uploaded (20)
Score card at Commsday Australia
- 1. Fight back against the ISP Scorecards Thomas Vasen Subscriber Experience Evangelist
- 2. 600 customers – 242 employees (35% R&D): 6 11 18 31 43 50 5 7 13 12 7 15 2010 2011 2012 2013 2014 2015 Existing New Customer Growth TIER-1 SERVICE PROVIDERS 15+ YEARS OF SOFTWARE DEVELOPMENT IN NETWORKS & APPLICATION AWARENESS Corporate Snapshot: Private company with the backing of Francisco Partners – a $10B+ portfolio HQ R&D R&D TAC TAC Confidential: Not for distribution without written consent of Procera Networks.
- 4. Anyone seen this before? Interesting, but not actionable
- 5. What about this one?
- 6. Or this?
- 7. OTT Providers are winning the PR WAR on performance
- 9. FCC Network Neutrality “With respect to network performance, we adopt the following enhancements: The existing transparency rule requires disclosure of actual network performance. In adopting that requirement, the Commission mentioned speed and latency as two key measures. Today we include packet loss as a necessary part of the network performance disclosure. We expect that disclosures to consumers of actual network performance data should be reasonably related to the performance the consumer would likely experience in the geographic area in which the consumer is purchasing service. We also expect that network performance will be measured in terms of average performance over a reasonable period of time and during times of peak usage.” Source: page 73-74 of FCC-15-24A1, Emphasis Added
- 10. Consumer’s Choice on Quality Airline Hospitality Restaurants
- 11. Not for Broadband Download 80.7 Mbps Upload 8.4 Mbps
- 12. Subscribers think Applications Each with different network quality requirements
- 13. Show quality per Application Type
- 14. Three ingredients to broadband quality 1. Throughput/Bandwidth/Speed 2. Latency 3. Packet Loss
- 15. Throughput/Bandwidth/Speed Application Type Throughput Needs Web Needs short bursts of download performance Video Sustained throughput delivers good quality Social Media Needs short bursts of download/upload performance Gaming Most games do not require high bandwidth Upload Sustained bursts of upload performance Download Sustained bursts of download performance Voice Low throughput requirements Very dependent on the sampling frequency!
- 16. Latency Application Type Throughput Needs Web High latency leads to slow page load times Video Not usually a concern except for initial loading of video Social Media High latency can slow interactive sharing experience Gaming High latency leads to lag in real-time games Upload N/A Download N/A Voice High latency leads to poor voice experience
- 17. Packet Loss Application Type Throughput Needs Web Packet Loss can lead to slow page load times. Video Less sensitive to loss unless it affects throughput Social Media Packet Loss can slow interactive sharing experience Gaming Packet Loss leads to lag in real-time games Upload N/A Download N/A Voice Some loss can be tolerated, high loss leads to perceived latency
- 18. Different collection approaches… Active Probes — Can affect the actual performance of network, especially during peak hours Sampling — Does not provide consistent data for entire footprint All subscriber traffic, all the time — Meets regulator guidance: • Actual customer experience • In the actual geographic area • During Peak Usage
- 19. Measuring all traffic, all the time enables an operator to create their own ScoreCard
- 20. Putting it All Together Application Type Throughput Latency Loss Web Needs short bursts of download performance High latency leads to slow page load times Packet Loss can lead to slow page load times. Video Sustained throughput delivers good quality Not usually a concern except for initial loading of video Less sensitive to loss unless it affects throughput Social Media Needs short bursts of download/upload performance High latency can slow interactive sharing experience Packet Loss can slow interactive sharing experience Gaming Most games do not require high bandwidth High latency leads to lag in real-time games Packet Loss leads to lag in real-time games Upload Sustained bursts of upload performance N/A N/A Download Sustained bursts of download performance N/A N/A Voice Low throughput requirements High latency leads to poor voice experience Some loss can be tolerated, high loss leads to perceived latency
- 21. “Graded” Quality Performance Score Subjective Experience A Awesome experience B Almost perfect, but some slight impairments noticed C Good experience but noticeable impairments D Usable with frustrating impairments E Really poor F Unusable
- 23. Case Study
- 24. Captured in the Core
- 25. Break down by Perspectives Device Location Subscriber Tier
- 30. Impact
- 31. In summary – 3 use-cases • Reporting to regulators on network performance with actual subscriber metrics • Marketing to subscribers — Targeted service offerings “Good for Gaming!” — High Value SLA-based offerings — Scoring Service Plans by performance • “An A for Video” — Promoting network capabilities • Improving the network where deficiencies occur or vs
Editor's Notes
- Hello, I’m Thomas Vasen, Subscriber Experience Evangelist at Procera. Today I will be bringing you reasoning on how operators should change how they measure the quality their network is capable of delivering in order to fight back the OTT players’ Scorecards. And I’ll bring you’re a case study how this was done successfully.
- In a full customer life-cycle, from finding a good deal, setting up the service towards actually using it, the incidental Support case and the monthly billing cycle – your Usage of the service has the most frequent and lasting interactions. This is why Operators should make sure they can deliver a good quality of service on their network that customers expect. With the Shift from Voice to DATA it also changes how often we “experience” the services. The Always-On lifestyle we have now is putting more stress on operators to be on their toes. Uptime is important! Availability / Accessibility is KEY!
- And if they don’t track this OTT Applications will. Who has seen this before? Netflix launched their own quality indicator for subscribers so they can see what quality they can expect. But this provides a very narrow view, does not take subscription types in to account and does not provide any actionable data points for the operator to improve their service upon.
- If you think that Netflix’s scorecard has little reach... Consider that many OTT services are building similar report cards. Here is one from YouTube, you can conveniently compare the video quality per operator in your area.
- And the old-timer Speedtest.net - they keep track of your customers' active testing. The problem with this is that users typically do their test twice: once they get the service – test for the first time – and when they have problems. The speeds published here are not an accurate representation of the services you provide!
- OTT Providers are Winning the Public Relations WAR on Performance
- BUT, speed is not an accurate representation of Quality – there are many more factors to consider.
- The FCC puts it clearly in their latest net neutrality regulation. This approach is copied everywhere. - The Commission mentioned speed and latency as two key measures. Today we include packet loss as a necessary part of the network performance disclosure --> 3 kay parameters - Also should be reasonably related to the performance the consumer would likely experience in the geographic area --> location specific - performance over a reasonable period of time and during times of peak usage. --> peak hour
- In most industries Consumers have a choice in selecting the quality they want. In the Airline industry you can buy Economy, Business or First class tickets. When choosing a Hotel you can select exactly how many star-ratings you want, and restaurants are grouped and rated by their type of food and quality. You can choose Fast Food or pick Michelin Stars.
- But for fixed and mobile broadband there only seems to be Speed and Volume. Once coverage is taken out of the equation: that's It. Not a very clear way to differentiate, and not very easy to understand for the subscribers.
- Subscribers think in OTT Applications. But each application has very different requirements on the networks to deliver a good quality of experience. And the impact of the network quality can be huge.
- Instead Operators should understand the quality they can deliver and categorize it by applications commonly used by subscribers. For instance these 7 easy to understand categories: Web Surfing Streaming Video Social Media Real-Time Gaming Uploads Downloads and Voice Applications Now let's have a look at how to do that.
- To do that you don’t have to dive deep. You just need to make sure you measure everything. Measure these 3 key parameters of all traffic flows, and map the results to the categories.
- READ SLIDE BUT it's very dependent on the sampling frequency! - 5-minute and 15-minute samples are averages --> Not very effective at detecting individual user level - 60-sec usage analysis with 1 second level granualrity tells much different story Peaks and valleys emerge at the individual user level, but many sessions will be missed. - Sampling sub-1sec is the answer. 250ms is a good balance between averaging bursts and capturing realized throughput Throughput overall is a good thing but it doesn’t tell a complete picture
- READ SLIDE
- READ SLIDE
- There are generally 3 ways to collect data Active Proves - effects performance, especially during peak. Only provides view from probe’s location. Passive Sampling – Provides only a partial view, lacks consistency All Traffic, All Subscriber, All the time Makes data actionable and meets regulator’s guidance on requirements to Secure the Actual experience, During Peak Usage.
- Measuring All Traffic, All the Time Enables an Operators to create their own ScoreCard
- Putting them all together
- And on different levels and combinations of resutlts measured you can now assign a grade. This makes is very easy to understand for consumers.
- Operators can now present their ScoreCard in a simple to understand format scaling from A to F. A "C" in Web Surfing likely means that there are some issues with packet loss. Social Media and especially Real-Time Gaming are very latency sensitive, and their scores are adjusted to the network’s capabilities accordingly.
- Let's have a look at a case study. This is from a fixed operator who wishes to stay anonymous. They where listed #10 on on the Netflix index.
- The data is typically captured in the core network, or close to the edges towards the interconnect points. This means very few measurement location are needed, while the solution is still able to separate traffic per access network through the use of reference data and source addressing schemes.
- Because an Overall Score for the network is great information, but still not actionable. The operator therefore groups the scores measured by many different perspectives, such as by Device, Location or Topology (such as RAN location, Wifi Access point or DSLAM) and Subscriber-Tier. This enables the operators to identify where problems occur, or what customer groups are impacted and act upon them.
- On the overall ScoreCard level degradations start to emerge during peak hour. The operator does not think their network is full, but yet Web browsing, Video and Gaming are impacted.
- Faulty load balancing
- This helped but is not enough to climb the Netflix ladder. Subscriber's perspective is needed
- Upsell strategies can be implemented
- Operator Tortoise.com has move up the scale and is providing a better service to those customers who care about it.
- In summary - 3 use-cases for an operator ScoreCard Reporting to regulators on network performance with actual subscriber metrics Marketing to subscribers Targeted service offerings “Good for Gaming!” High Value SLA-based offerings Scoring Service Plans by performance “An A for Video” Promoting network capabilities Improving the network where deficiencies occur.
- A new perception of what Quality for a Mobile or Fixed Broadband service is is needed.