1. It’s Not the Cost, It’s the Quality!
Ion Stoica
Conviva Networks and UC Berkeley
1

2. A Brief History
! Fall, 2006: Started Conviva with Hui Zhang (CMU)
! Initial goal: use p2p technologies to reduce
distribution costs and improve the scale
! Slowly, realized our customers (content premium
producers & aggregators) value more quality than
cost
! Today: maximize distribution quality, distribution
management, and provide real-time analytics
2

3. Where is the Data Coming From?
! Content Providers and Aggregators
! CDNs
3

4. 4
Trends

5. Trends: CDN Pricing
! CDN pricing has decreased x1.5-2 every year
over the last 5 year
5
0
5
10
15
20
25
30
35
40
45
2006 2007 2008 2009 2010
cents/GB

6. Trends: Streaming Rate for Premium
Content
! Average streaming rate has increased 20-40%
every year
6
0
200
400
600
800
1000
1200
1400
1600
2006 2007 2008 2009 2010
Kbps

7. Trends: Per-hour Streaming Cost
! Per-hour streaming cost has decreased 15-35%
every year
7
0
1
2
3
4
5
6
7
8
9
10
2006 2007 2008 2009 2010
cents/hour

8. HTTP Chunking
! Trend accelerated by switching from proprietary
streaming technologies (e.g., Adobe’s FMS) to HTTP
Chunking:
 Move Networks (2005)
 Apple (2008)
 Microsoft (2008/2009)
 Adobe (2010, 2nd half)
8

9. How Does HTTP Chunking Work?
9
Viewer
CDN1
Viewer
Viewer
CDN2
ISP A ISP B
…
origin
http cache

10. HTTP Chunking Advantages
! Chunks: immutable, relative large objects
(hundreds of KB)
 Great for caching
! Leverage existing HTTP infrastructure
 CDNs
 ISP deployed caches
 Enterprise http proxies
! Low cost and high scale
10

11. What Does this Mean?
! Ad supported premium content
 CPM (cost per thousand of ad impressions) for
premium content has reached: $20-$40
 One ad covers one hour of streaming!
! Paid content
 $0.99 episode, distribution cost < 3%
! Subscription based premium content
 Distribution, usually a few percents of total cost
 It costs $1.6 per month to stream content to an user
watching 2 hours per day
! Production & rights costs dominate
11

12. 12
Quality Matters

13. Quality Matters
! Better quality
 Increase viewing time  more ad opportunities
 Increase retention rate
 Protect brand
! Quality
 Join time
 Buffering ratio
 Rendering quality
 Streaming rate
13

14. Analysis
! Load:
 Four channels of a premier video-on demand (VoD)
content producer
 Four days
 Number of sessions (views): 1,176,049
 A large live event: ~250,000 concurrent viewers
! Metrics
 Content length distribution
 Viewer Hour Loss (VHL): number of viewer hours lost
due to quality issues

15. VoD Object Length Distribution
15
0
50
100
150
200
250
300
10
40
70
100
130
160
190
220
250
280
310
340
370
400
430
460
490
520
550
580
610
640
680
750
880
1240
1330
2540
2570
2600
2640
4040
5410
5550
Object Length
Short clips:
[2min, 3min]
Medium clips:
[9min, 11min]
Full Episodes:
[42min, 45min]
NumberofObjects

16. Quality Metrics
! Buffering Quality (BQ):
PlayingTime/(PlayingTime + BufferingTime)
 Rendering Quality (RQ):
RenderingRate/EncoderRate
 Good session
 BQ > 95%
 RQ > 60%

17. Analysis Underestimates Quality
Impact
! For most analysis use BQ only
 RQ only a small part of quality issues due to low
bit rate (500-700Kbps)
! Ignore connection failures

18. Short Clip (2-3min) Analysis
18
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50
AveragePlayingTime(Minutes)
Quality

19. Short Clip (2-3min) Analysis
19
0
50000
100000
150000
200000
250000
300000
98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50
NumberofSessions
Quality

20. Viewer Hour Gain
! D : Average duration of sessions with high quality
(0.98 <= quality < 1)
! Dq : Average duration of sessions with quality = q
! Nq : Number of sessions with quality = q
! Viewer hour gain for sessions with quality q
Nq x (D – Dq)
! Total viewer hour gain
∑q Nq x (D – Dq)
20
Viewer hour loss for 1-2 minute clips:

21. Medium Clip (9-11min) Analysis
21
0
1
2
3
4
5
6
7
8
98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50
AveragePlayingTime(Minutes)
Quality
Viewer hour loss for 9-11min clips:

22. Full Episodes (42-45min) Analysis
22
0
5
10
15
20
25
98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50
AveragePlayingTime(Minutes)
Quality
• Viewer hour loss for episodes:
• Viewer hour loss for all content:

23. Large Scale Live Event
23
0
5
10
15
20
25
30
35
98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50
AveragePlayingTime(Minutes)
Quality
Viewer hour loss:

24. Large Scale Live Event: Engagement Funnel
24
1 0.99
0.8624
0.7364
0.6602
0.5509
0.4357
0.3639
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
All 0-0 0-1 1-3 3-5 5-10 10-20 20-30
FractionofSesions
Session Duration (Mins)
0<=Q<=0.5
0.5<=Q<=0.75
0.75<=Q<=0.9
0.9 <= Q <= 0.95
0.95<=Q<=0.98
0.98<=Q<=1.0
Quality = 1.0
Continuing
Half people leave due to quality
issues

25. Another Case Study: Live Event
25
Total sessions 151,980
Unique viewers 73,942
Sessions per viewer 1.9
Total viewer hours 58,436
0
5000
10000
15000
20000
7:00PM
7:15PM
7:30PM
7:45PM
8:00PM
8:15PM
8:30PM
8:45PM
9:00PM
9:15PM
9:30PM
9:45PM
10:00PM
10:15PM
10:30PM
10:45PM
11:00PM
11:15PM
11:30PM
11:45PM
12:00AM
12:15AM
12:30AM
12:45AM
1:00AM
Peak Concurrent Views
Quality Engagement
Total views 151,980 25 minutes
Failed views 13,815 (9%) 0 minutes
Quality impacted
views
21,584 (14%) 16 minutes
Good views 116,581 (77%) 27 minutes
Unique viewers 75,328 48 minutes
Failed viewers 1,386 (2%) 0 minutes
Quality impacted
viewers
14,309 (19%) 30 minutes
Good viewers 59,633 (79%) 51 minutes
Total viewer hours 58,436 hours
Lost viewer hours 5,134 hours
(9%)
Viewer with poor quality watch 41%
less minutes!

26. Does High Bit Rate Video Help?
! Comparing Engagement of low and high bitrates
 Viewers watch longer on average on 1500Kbps
0
5
10
15
20
25
30
35
minutes
Average Play Time (minutes)
500 kbps
1500 kbps
10-15% increase

27. Summary
! Quality impact:
 BQ can impact viewer engagement by up to 40%
 Higher bit-rates can increase viewer engagement by
up to 15%
! Engagement loss due to quality issues: between
4 and 30%
 Even a 4% improvement, may offset distribution
costs
 Ignore other quality issues, like connectivity and
media failures
27

28. 28
Root Cause Analysis

29. Viewers vs. Buffering Quality
29
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0
50000
100000
150000
200000
250000
0:00:00
0:15:00
0:30:00
0:45:00
1:00:00
1:15:00
1:30:00
1:45:00
2:00:00
2:15:00
2:30:00
2:45:00
3:00:00
3:15:00
3:30:00
3:45:00
4:00:00
4:15:00
4:30:00
4:45:00
5:00:00
5:15:00
5:30:00
5:45:00
6:00:00
6:15:00
Viewers Average Buffering Quality %
Light Period Heavy Period
What causes quality issues?

30. Root Cause Analysis
! Root cause a quality issue to:
 Viewer machine (CPU)
 Last mile + ISP (Autonomous System Number)
 CDN
! Note:
 Cannot differentiate between edge and core ISPs
 Use only passive measurements, no IP traceroute
Viewer
Last Mile
+
ISP
CDNPeering
ISP

31. Metrics and Definitions
! Quality metrics
 Buffering quality (BQ)
 playing time/(playing time + buffering time)
 Rendering quality (RQ)
 rendering frame rate/encoded frame rate
! Session classification:
 Good: (BQ >= 95%) AND (RQ >= 60%)
 Low BQ: (BQ < 95%)
 Low RQ: (BQ >= 95%) AND (RQ < 60%)

32. Methodology: Root Causing Viewer
Machine
! CPU likely to be the issue when:
 Rendering quality low
 Buffering quality high
! Conclude CPU is the issue when session’s
 RQ < 60%
 BQ > 95%

33. Good
Low BQ
Low RQ
Quality Issues: Light Period
74%
21.5%
4% (CPU issue)
Network/CDN
issues

34. Good
Low BQ
Low RQ
Quality Issues: High Period
62%
6.2% (CPU issue)
31.5%
Network/CDN
issues

35. Explaining Buffering Issues
! Assume buffering quality issues are either due to:
 CDN, or
 ISP
! Recall: a session has buffering quality issues if
 BQ < 95%

36. Methodology: Root Causing CDN (1/2)
! Viewers connected to same ASN but using two CDNs
! Intuition: if quality experienced by CDN 1 viewers is
significantly lower than of CDN 2 viewers for same ASN,
CDN 1 has quality issues
Last Mile
+
ISP
CDN 1Peering
ISP
CDN 2
Peering
ISP

37. Methodology: Root Causing CDN (2/2)
! Select all ASNs who have more than 50
sessions for each CDN
• If difference between quality of viewers in CDN1 and
CDN2 for same ASN is > 10%
 Lower quality CDN is root cause at current time

38. Methodology: Root Causing ASN/ISP
! Two CDNs:
 Conclude ASN A has quality issues if ASN A’s viewers
connected to either CDN1 or CDN2 experience “bad
quality”
 Average quality of viewers connected to other ASNs
higher
! One CDN
 ASN A’s viewers connected to CDN have much lower
quality than the average quality of viewers connected
to CDN

39. Buffering Quality: Light Period
Unqualified
CDN
ISP
Uknown
46%
33%
16%
5%

40. Buffering Quality: Heavy Period
40
Unqualified
CDN
ISP
Uknown
36.7%
39.7%
15.4%
8.2%

41. Some Findings
! Most of ASNs who had quality issues were
enterprise ASNs
 Expected given that the large scale event was during
the workday
 One ASN had 44% buffering quality!
! No CDN was uniformly bad
 (see next)

42. CDN Comparison
! Quantify quality difference between CDNs
! Methodology:
1. Select all ASNs which have more than 50 sessions
on both CDNs
2. Compute average quality for CDN1 and CDN2
viewers per ASN
3. Order ASNs by difference in quality between CDN1
and CDN2

43. Internet delivery is more variable
than realized…
! Content Delivery
Networks all
have problems
sometime
! Even in the same
viewer session
the best quality
changed many
times during the
event
CDN 1 was best
CDNs were even
CDN 2 was best

44. Summary
24-38%ofTotalSessionshaveQualityIssues
Quality Issues
Classification
Solution
CDN
(7-12% of total sessions)
Resource switching
End-Host CPU
(4-6% of total sessions)
Bit-rate switching
ISP
(2-3% of total sessions)
Localize traffic, bit-rate switching
Unqualified
(9-11% of total sessions)
Mitigated by above
Unknown
(1-4% of total sessions)
N/A

45. Conclusions
! At least for premium content
 Reducing cost is important, but…
 … improving quality is even more important
! P2P can play an important role
 Localize traffic
 Highly robust to source failures
! Great opportunity
 Adobe has announced full p2p support for Flash
Player 10.1
 No need for client download!
45