This slide telling what "adaptive streaming" is. In the beginning, it explains how content(media) is prepared to fit adaptive needs, then talking about yapi.js - which is the web player used by KKTV(VOD service) in Taiwan.

1. ADAPTIVE MEDIA PLAYER
BASED ON HTML5
陳建⾠ jessechen
yapi.js

2. surf, sb, football, draft beer, kktv
jessechen陳建⾠
http://blog.hellojcc.tw

3. OTTVOD SERVICE
台⽇韓中
超⼤⽚庫
⼤數據精準推薦
離線播放
投放⾄電視

5. PLAYBACK EXPERIENCE
ON WEB
WITH HTML5

6. 影⽚
yapi .js
（台）

7. WHAT IS BEING PLAYED
media

8. analog-to-
digital adapter
digital raw data
PREPARE MEDIA
1. Make it digital
analog signal

9. H.264 / H.265
Encode
PREPARE MEDIA
2. Encode video file
Raw data .mp4 / .ts file
Easier for storage and transference

10. server holds
media
VINTAGE WAY
Player get it and
play
single file
super slow

11. server holds
media
EFFICIENT WAY
Player get part
of video and
start play
streaming
(progressive download)
Keep requesting
rest part of
video during
playback
inflexible

12. PREPARE MEDIA
3. Fit adaptive need
.mp4 / .ts file
more storage needed but much flexible
different bitrate
+
+
fragments
manifest
file

13. MANIFESTTELLS
- adaptions: video, audio or subtitle
- available bitrates
- how file is fragmented
- encryption

14. ADAPTIVE STREAMINGTYPES
SS
Smooth Streaming
HLS
HTTP Live Streaming
DASH
Dynamic Adaptive
Streaming through HTTP

15. .mp4 / .ts file different bitrate
+
+
fragments
manifest
file
Raw data
imaging there are thousands of media needs to handle

16. CONTENT OF KKTV IS
PREPARED WITH
MASS
https://github.com/KKBOX/mass
https://goo.gl/6TCmWqintroduction by Jinkuen:

17. download
adaptive streaming
client
yapi.js
media
manifest
file
adaptive
algorithm
fragments

18. adaptive algorithm decides
which quality to download

19. • network status
• how long does buffered duration left
algorithm considering…

20. adaptive
starting lag
buffering
times / duration
overall bitrate

21. a/v sync
decode
decrypt
‘playback’
client
yapi.js
media
manifest
file
fragments

22. decrypt
encryption + decryption

23. H.264 / H.265
Encode
ENCRYPTION
Raw data .mp4 / .ts file
encryption
implemented
here

24. CLIENT SIDE DECRYPTION
• Client get a license instead of a key
• A blackbox/sandbox get the key by processing license
• That blackbox/sandbox provide key for decryption and
directly output to display
• This solution called DRM(Digital Right Management)

25. DRM ON BROWSER
• For browser, that blackbox called CDM
(Content Decrypt Module)
• Each browser support different DRM
context - “a blackbox or sandbox get the key by
processing license”

26. DRM ON BROWSER
Widevine FairplayPlayready Primetime
Widevine

27. INTRODUCE EME
• Even though browser support its own DRM,
W3C defines a EME spec, in order to expose same
api to use
• prefixed api was implemented on earlier version of
chrome(smart tv)
Encrypted Media Extension

28. YAPI’S PROTECTION LOGIC
• implement protection on client side through EME
• give what DRM server needs to retrieve license
• deal with different browsers (versions)

29. EME
CDM
attempting to
play encrypted
media
get ‘challenge’
DRM
license
server
request with challenge
get license
yapi
provide license
for CDM to
decrypt content
EME
CDM
PROTECTION LOGIC FLOW

30. a/v sync
decode
html5 media element + MSE

31. MEDIA ELEMENT
var videoNode = document.createElement(‘video’);
videoNode.src = VIDEO_URL;
With html5 media element, you can play single video source easily

32. single source multiple source
+
+

33. MSE
“ MSE (Media Source Extension) extends HTMLMediaElement to allow
JavaScript to generate media streams for playback.
Allowing JavaScript to generate streams facilitates a variety of use cases like
adaptive streaming and time shifting live streams.“
media source extension

34. var video = document.createElement(‘video’);
video.src = VIDEO_URL
MEDIASOURCE IS A ‘SOURCE’
set ‘src’ attribute of video element to an url pointed to media
source
new window.MediaSource();window.URL.createObjectURL(ms);

35. SOURCE BUFFER
sourceBufferVideo = ms.addSourceBuffer(VIDEO_CODEC);
sourceBufferAudio = ms.addSourceBuffer(AUDIO_CODEC);
// get stream buffer via network
sourceBufferVideo.appendBuffer(buffer);
// sourcebuffer provides buffer info after append complete

36. BUFFER INFO
var buffered = sourceBuffer.buffered;
buffered.length; // how many discontinuous buffered time range
buffered.start(0); // first buffer start time
buffered.end(0); // first buffer end time
get buffer information from buffered attribute

37. MSE EXTENDS MEDIA ELEMENT
• MSE focus on providing stream buffer to media
element
• playback behavior still hold by media element
e.g play, pause, seek

38. a/v sync
decode
‘playback’
client
yapi.js
media
manifest
file
fragments
decrypt

39. streaming
adaptive
protection / EME
mediaElement / MSE
client
yapi.js

40. BETWEEN MODULES
1. hold reference
problems
• dependency
streaming adaptive

41. BETWEEN MODULES
2. a parent module
problems
• defining ‘parent’
• multiple parent modules?
streaming
adaptive
parent
parent？

42. BETWEEN MODULES
3. event system
• modules are parallel
• semantic event name
indicates current ‘state’ of
player
streaming
adaptive
protection / EME
mediaElement / MSE
eventBus

43. streaming
adaptive
MSECtrl
eventBus
eventBus
eventBus
eventBus

44. function downloadFragment() {}streaming
adaptive
eventBus
eventBus
MSECtrl eventBus
function calculateQuality() {}
function appendToSourceBuffer() {}
function onDownloaded() {
// notify FRAGMENT_DOWNLOADED
}
function onAppended() {
// notify FRAGMENT_APPENDED
}

45. register
eventBus.mapHandler(‘FRAGMENT_DOWNLOADED’, )adaptive.calculateQuality
eventBus.mapHandler(‘FRAGMENT_DOWNLOADED’, )MSECtrl.appendToSourceBuffer
eventBus.mapHandler(‘FRAGMENT_APPENDED’, )streaming. downloadFragment

46. {
}
FRAGMENT_DOWNLOADED: [ ,
],
adaptive. calculateQuality
MSECtrl. appendToSourceBuffer
FRAGMENT_APPENDED: [ ]streaming. downloadFragment
eventBus

47. function downloadNextFragment() {}streaming
adaptive
eventBus
eventBus
MSECtrl eventBus
function calculateQuality() {}
function appendToSourceBuffer() {}
function onDownloaded() {
eventBus.dispatch(‘FRAGMENT_DOWNLOADED’)
}
function onAppended() {
// notify FRAGMENT_APPENDED
}
dispatch event

48. eventBus
{
}
FRAGMENT_DOWNLOADED: [ ,
],
adaptive. calculateQuality
MSECtrl. appendToSourceBuffer
FRAGMENT_APPENDED: [ ]streaming. downloadFragment
invoke callbacks

49. function downloadNextFragment() {}streaming
adaptive
eventBus
eventBus
MSECtrl eventBus
function calculateQuality() {}
function appendToSourceBuffer() {}
function onDownloaded() {
eventBus.dispatch(‘FRAGMENT_DOWNLOADED’)
}
function onAppended() {
eventBus.dispatch(‘FRAGMENT_APPENDED’)
}
dispatch event

50. eventBus
{
}
FRAGMENT_DOWNLOADED: [ ,
],
adaptive. calculateQuality
MSECtrl. appendToSourceBuffer
FRAGMENT_APPENDED: [ ]streaming. downloadFragment
invoke callbacks

51. works with app level
yapi.js
app
ui
client
api

53. works with app level
yapi.js
app
ui
client
events
api

54. MEDIA EVENTS
http://www.w3.org/2010/05/video/mediaevents.html
Dispatched by HTML5 media element, notifying playback process

55. YAPI PROXY/PATCH EVENTS
Media
Element
event event with same name
yapi.js

56. USEFUL MEDIA EVENTS
loadstart: Indicate loading media begins, this fires when setting src attribute
loadedmetadata: while element has basic info of playback, e.g duration
timeupdate: while current time of playback is ticking
seeking/seeked: while conducting seek
ended: playback ends
play/playing: playback resume from other status to playing

57. ADDITIONAL EVENTS
loadedmanifest: after manifest is loaded/parsed
bitratechanged: when bitrate is changed
enableabr: when adaptive activation changed
buffering: when playback pending
cuechanged: webvtt subtitle cue changed (in and out)

58. yapi.load(MANIFEST_URL); // exposed api
yapi.addEventListener(‘playing’, onPlaying);
function onPlaying() {
// app logic
// or ui reaction, e.g showing playing status ui
}
// onPlaying invoked when playing
app
yapi ui

59. LESSONS

60. 1. test case helps

61. 2. handle errors

62. 3. boss wants ‘numbers’

63. streaming
adaptive
protection / EME
mediaElement / MSE
client
yapi.js
statistic
playing smoothly is not
enough
you need to tell how
good it is

65. LIVE DEMO
https://app.kktv.me

66. mse spec: https://goo.gl/L3HPKP
media event sample: https://goo.gl/BsrH18
MDN media element: https://goo.gl/JsojcN
HTML MediaElement doc: https://goo.gl/Y3lbO0
dash.js: https://goo.gl/xAm2Ve

67. THANKYOU
KKTV is hiring
web / android / iOS / backend developers
jessechen@kktv.me