Delivery of video depends on a complex streaming ecosystem with many points of failure. For example, a publisher may fail to upload certain video assets; an ISP may experience congestion at several points in its network; or a home user may have a poor WiFi signal to their device. Having gathered data on video quality from many kinds of playing devices across the United States, Conviva is able to attribute quality deteriorations to the different parts of this ecosystem. In this session, you’ll learn about the nature and scope of the data, Conviva’s use of machine learning models in fault attribution, their use of Apache Spark and Databricks, and their results.

1. Oleg Vasilyev, Henry Milner, Wensi Fu, Oleg White
Assigning Responsibility
for Deteriorations in
Video Quality

2. …is a video experience management
platform that maximizes viewer
engagement.
We provide quality metrics that give a
comprehensive view into online video
businesses.

3. Devices & OVPs
Internet Video Streaming is Hard
Many parties, many paths, but no E2E owner
ISPs and Networks CDNs Publishers
CDN
Cable/DSL ISP
Backbone Network
CDN
Wireless ISP
Encoder
Video Origin &
Operations
Home WiFi

4. Devices & OVPs
Internet Video Streaming is Hard
Many parties, many paths, but no E2E owner
ISPs and Networks CDNs Publishers
CDN
Cable/DSL ISP
Backbone Network
CDN
Wireless ISP
Encoder
Video Origin &
Operations
Home WiFi
Publisher
control
ISP
control

5. Devices & OVPs ISPs and Networks CDNs Publishers
CDN
Cable/DSL ISP
Backbone Network
CDN
Wireless ISP
Encoder
Video Origin &
Operations
Home WiFi
100
publishers
1,000,000
unique
videos
(“assets”)
100 CDNs100,000 nodes in internal ISP
network
1,000,000 IP
addresses
1,000 unique
devices
The ecosystem is big.
…in 10,000,000 video sessions from one week, at one US ISP

6. Viewers are expecting TV-like quality (or better)
QoE is Critical to Engagement
For both video and advertisement businesses
HOW LIKELY ARE YOU TO WATCH
FROM THAT SAME PROVIDER AGAIN?
33.6%
VERY
UNLIKELY
24.8%
UNLIKELY
24.6%
UNCHANGED
58.4%CHURN RISK
6.6%
VERY LIKELY
8%
LIKELY
-3
-8
-11
-14
-16
2011 2012 2013 2014 2015
ENGAGEMENTREDUCTION
(IN MINUTES)
WITH 1% INCREASE IN BUFFERING
MINS
MINS
MINS
MINS
MINS
Source: Conviva, “2015Consumer Survey Report”

7. Devices & OVPs ISPs and Networks CDNs Publishers
CDN
Cable/DSL ISP
Backbone Network
CDN
Wireless ISP
Encoder
Video Origin &
Operations
Home WiFi

8. Quality impact
Replace internal
ISP fiber node
No change
0.1% of session
spent buffering
2% of session
spent buffering

9. Quality impact of
this fiber node on
session 3
Quality impact of
this fiber node on
session 2
Quality impact
0.1% of session
spent buffering
2% of session
spent buffering
-
Quality impact of
this fiber node on
this session
=
Quality impact of this
fiber node
Quality impact of
this fiber node on
session 1=
𝚺

10. Quality impact
Replace internal
ISP fiber node
No change
0.1% of session
spent buffering
2% of session
spent buffering
???

11. Typically better
home WiFi
Confounding factors
0.1% of session
spent buffering
2% of session
spent buffering
Different fiber node
Actual fiber node
Actual effect
Effect of confounder

12. Devices & OVPs ISPs and Networks CDNs Publishers
CDN
Cable/DSL ISP
Backbone Network
CDN
Wireless ISP
Encoder
Video Origin &
Operations
Home WiFi
100
publishers
1,000,000
unique
videos
(“assets”)
100 CDNs100,000 nodes in internal ISP
network
1,000,000 IP
addresses
1,000 unique
devices
Many potentially-important factors
…and confounding factors!

13. Typically better
home WiFi
Fixing confounding
0.1% of session
spent buffering
2% of session
spent buffering
Different fiber node
Actual fiber node

14. Typically better
home WiFi
Fixing confounding
0.1% of session
spent buffering
2% of session
spent buffering
Different fiber node
Actual fiber node
Estimate both
effects

15. Estimating quality impact
Fiber Node = #1
(a good FN)
Fiber Node = #567
(actual FN)
1.7% of session
spent buffering
2.5% of session
spent buffering
Model
- IP: 1.2.3.4
- Fiber Node: #567
- Asset: “Glee”
1.7% of session
spent buffering
2.5% of session
spent buffering
-
Estimated quality
impact of this fiber
node on this session
=

16. Devices produce short buffering

17. Devices produce short buffering

18. Device-level issues are responsible
for most short buffering episodes

19. Good and bad IP addresses

20. From worst PCD, typical week:

21. From worst PCD, typical week:

22. Worst assets, typical week:
Only 6 really bad assets responsible for strong
deterioration of their sessions regardless of other factors.
Of these, 4 are sport programs and 2 are obscure regularly
scheduled foreign programs.

23. Modeling video quality
random forests
boosted trees
neural networks
GLMs
Today’s results

24. Model for rebuffering ratio
Response: Rebuffering ratio (R)
(buffering time / (buffering time + playing time))
Categorical features:
IP address (I)
Fiber Node (N)
Service Group (S)
Publisher (P)
Asset (A)
CDN (C)
Device (D)
Live / VoD
More features:
Time, day of week, asset length.

25. Time or no time?
Time is strongest or one of strongest features.
Big game, popular show – many sessions suffer, regardless of IP and
device or even CDN.
Time makes model more precise.
But: time steals effect from big nodes such as Asset and Publisher.
Similar question: Bitrate or no bitrate?

26. Model versions, practical issues
Preference: Spark cluster of 10-30 nodes, each node 4-8 cores and
30GB memory. Hopefully 1-3 hours to process 1-3 weeks of data of big
ISP, whole US.
Nodes as embedded features. Boosted trees on Spark. Whole US. Learn
from 1-3 weeks, apply to last week.
Nodes as one-hot encoding. Random forest on Spark. Each
geographical area is processed separately.

27. Model versions, practical issues
Nodes as a trainable embedding first layer. Neural network on single
Spark node with Tensorflow. Each geographical area separately.
Embedding size is same for all kinds of nodes.
Embedding size is ~ log of node's dictionary size.
One or two hidden layers above the embedding layer.
Adagrad or any other Ada-like optimizer performs better than no-Ada.
Slow improvements beyond fast mediocre accuracy.

28. Finding a “good” node
Effect of a node = Model( real features ) - Model( features with good node )
In case of trainable embedding – iterative finding of good nodes:
good = nodes with lowest avg label
while set of good nodes is changing:
for each session:
Effect = Model(real) - Model(good)
good = nodes with lowest avg effect
Overlap of set of good nodes with
the set from previous iteration -
typically stabilizes in 2-3 iterations.

29. Sessions are affected by ...
If effect is linear, this would be like ART in 3D tomography

30. Thanks to Spark and Databricks
for making it easy for us

31. Thank You.
Contact information:
ovasilyev@conviva.com
hmilner@conviva.com
We Are Hiring!
www.conviva.com/our-team/careers