The document details the development of an HTML5 video streaming player, highlighting key processes such as video encoding, adaptive streaming, and digital rights management (DRM). It outlines the architecture of the player, focusing on features like manifest parsing, content protection, and playback of adaptive bitrate segments. Future goals include enhancing DRM support across various platforms and integrating additional functionalities like live streaming and ad support.

1. HOW DID I CREATE A HTML5
VIDEO STREAMING PLAYER
and what I’ve learned
KKBOXVideo
陳建辰 jessechen

2. AGENDA
• Some beforehand knowhow
• Why developing this player
• HTML5 extensions
• Introduce our player
• Future of this player

3. SOME BEFOREHAND
KNOWHOW

4. analog-to-
digital adapter
digital raw data
PROCESS OF PLAYINGVIDEO
1. Make it digital
analog signal

5. H.264 / H.265
Encode
PROCESS OF PLAYINGVIDEO
2. Encode video file
Raw data .mp4 / .ts file
Easier for storage and transference

6. PROCESS OF PLAYINGVIDEO
3. Play it !
.mp4 / .ts file
a/v sync
codec decode
and
display
extract content
from container

7. WAIT,WHERE IS PLAYER?

8. WHAT PLAYER DOES
.mp4 / .ts file
a/v sync
codec decode
and
display
HERE
extract content
from container

9. THEY ARE ALREADY HANDLED
.mp4 / .ts file
HTML5
a/v sync
codec decode
and
display
extract content
from container

10. SO WHY DEVELOPING
THIS PLAYER?

11. OUR PLAYER FOCUS ON (1)
extract content
and a/v info
from container
.mp4 / .ts file
get files
in smart way
a/v sync
codec decode
and
display

12. HOW PLAYER GET FILE ?

13. server holds
.mp4
content
VINTAGE WAY
Player get it
and play
single file
slow and inflexible

14. server holds
.mp4
content
EFFICIENT WAY
Player get part
of video and
start play
progressive download
Keep requesting
rest part of
video during
playback
still inflexible

15. prepare
different qualities
of content
on server
MODERN WAY
adaptive streaming
Player plays
best quality
depends on
network status
Keep requesting
rest part of
video during
playback

16. HOW ABOUT WE SPLIT FILE
• split file into segments, described by manifest
• manifest tells
- available bitrates
- how file is fragmented
- other information, e.g encryption
+
+

17. knows how to
get fragments
ADAPTIVE STREAMING
IN SHORT
fragments
manifest
file

18. ADAPTIVE STREAMINGTYPES
SS
Smooth Streaming
HLS
HTTP Live Streaming
HDS

19. INTRODUCE DASH
Dynamic Adaptive Streaming through HTTP

20. PROS OF DASH
• open source, with industry standard, which means
more universal usage
• support 2 types of containers, the MPEG-4 file
format or the MPEG-2Transport Stream

21. GET FILES IN SMART WAY
• manifest parser - able to read different type of
manifest
• adaptive bitrate - able to decide which quality to
load

22. OUR PLAYER FOCUS ON (2)
.mp4 / .ts file
get files
in smart way
protection
logic
extract content
from container
a/v sync
codec decode
and
display

23. DRM ?
(DIGITAL RIGHTS MANAGEMENT)
Content provider /
DRM provider likes it
For end user

24. CLIENT SIDE PROTECTION
• Client get a license instead of a key from DRM
service
• A blackbox or sandbox get the key by processing
license
• That blackbox / sandbox decrypt content and
directly output it to display

25. H.264 / H.265
Encode
HOWTO PROTECT CONTENT
Raw data .mp4 / .ts file
DRM
implemented
here

26. PROTECTION LOGIC
• give what DRM server needs and retrieve license
• negotiate with browser in order to implement
protection on client side
• deal with different browsers
knows how to

27. NOW, LET’S STARTTALK
ABOUT BROWSER
<video>, MSE and EME

28. HTML5VIDEO ELEMENT
• <video></video> support progressive download
• For adaptive streaming, MSE is required

29. extract content
from container
WHAT ROLE MSE PLAYS
.mp4 / .ts file
MSE handles
a/v sync
codec decode
and
display
MSE handles

30. INTRODUCE MSE
• Given different bitrate segments and it helps
html5 video element to play smoothly
• SourceBuffer provide video and audio buffer
condition
Media Source Extension

31. EME
IS ABOUT PROTECTION

32. HOW BROWSER ADOPT DRM
• For browser, that blackbox called CDM
(Content Decrypt Module)
• Each browser support different DRM
context - “a blackbox or sandbox get the key by
processing license”

33. DRM ON BROWSER
Widevine FairplayPlayready Primetime

34. INTRODUCE EME
• Even though browser support its own DRM,
W3C defines a EME spec, in order to expose
same api for client
• prefixed api was implemented on earlier version
of chrome(smart tv)
Encrypted Media Extension

35. EME
CDM
provide context
from encrypted
content
get ‘challenge’
DRM
license
server
request with challenge
get license
provide license
for CDM to
decrypt content
player
EME
CDM
PROTECTION LOGIC FLOW

36. WHAT ROLE EME PLAYS
extract content
from container
.mp4 / .ts file
a/v sync
codec decode
and
display
decrypt content
in blackbox

37. INTRODUCE OUR PLAYER

38. OUR GOAL
• play not single file but sequence of segments,
with different bitrate, a.k.a adaptive streaming
• play protected stuffs, for content providers’ goods
a player able to

39. 影片
ya pi
yapi.js

40. DEVELOP PROCESS
make it work stable refactor

41. MAKE IT WORK FIRST
only 2 files in the very beginning

42. BE STABLE
• well structured
• modularized
• dependency management (dijon.js)
• consistent code style

43. MODULARIZED
api
adaptive
extension
fragment
manifest
protection
stats
stream
utils

44. UTILS
• Capabilities
• Debug
• ErrorHandler
• EventBus
• IntervalBus
• UrlModel
api
adaptive
extension
fragment
manifest
protection
stats
stream
utils

45. EXTENSION
• MediaSourceExtension
• SourceBufferExtension
• VideoModel
api
adaptive
extension
fragment
manifest
protection
stats
stream
utils

46. MANIFEST
• ManifestLoader
• ManifestExtension
• ManifestModel
api
adaptive
extension
fragment
manifest
protection
stats
stream
utils

47. STREAM
• StreamCtrl
• BufferCtrl
api
adaptive
extension
fragment
manifest
protection
stats
stream
utils

48. FRAGMENT
• FragmentCtrl
• FragmentLoader
• SegmentTemplate
api
adaptive
extension
fragment
manifest
protection
stats
stream
utils

49. ADAPTIVE
• AbrCtrl
• BandwidthRecorder
api
adaptive
extension
fragment
manifest
protection
stats
stream
utils

50. PROTECTION
• ProtectionCtrl
• ProtectionModel
• ProtectionRequest
• Playready / Widevine
api
adaptive
extension
fragment
manifest
protection
stats
stream
utils

51. STATS
• MetricsCtrl
• Metrics
• StatsCtrl
• BitrateHistory
api
adaptive
extension
fragment
manifest
protection
stats
stream
utils

52. REFACTOR
• review flow
• redefine api

53. APPLICATION FLOW

55. REDEFINE API
• use jsdoc to generate spec document
• define api, event and vo(value object)
• spec
• result

56. DEMO
Sample player

57. HOWTO MANAGE
THOSE MODULES

58. DEPENDENCY INJECTION
• not necessary in the beginning, but became very
important after it went much more complex
• use dijon.js as di framework

59. INTRODUCE DIJON.JS
• a system holds all dependencies, after mapping
modules to it
• an object get dependencies after injection

60. DEFINE DEPENDENCIES
// A dep
function A() {}
// B requires A
function B() {
this.a = undefined; // how dijon knows where to inject
}
// instantiate dijon
var system = new dijon.System();

61. 3 KINDS OF MAPPING
var a = new A();
system.mapValue(‘aValue’, a);
// system.getObject(‘a’) would return an ‘a’ instance
system.mapSingleton(aSingleton,A);
// system.getObject(‘aSingleton’) would return a singleton a
system.mapClass(bClass, B);
// system.getObject(‘bClass’) would return a new inited B

62. OUTLET MAPPING
// map outlet to make dependency work
// @param sourceKey {String} - the key mapped to system would be
injected
// @param targetKey {String|'global'} - the key outlet is assigned to
// @param outletName {String|sourceKey} - the name of property used as
an outlet
system.mapOutlet('aSingleton', 'bClass',‘a');
var b = system.getObject(‘bClass’);
// b.a is aSingleton

63. DIRECTLY INJECTION
var b = new B();
system.injectInto(b);
// b WOULDN’T have dependency singleton A
// b/c we only mapOutlet to key bClass of system
// map outlet to global
system.mapOutlet(‘aSingleton’,‘global’,‘a’);
system.injectInto(b); // this would work

64. AUTO MAP OUTLET
function B(){
this.aSingleton = undefined;
this.c = undefined;
}
function C(){}
// auto mapped outlet would assign to ‘global’
// and outlet name is the same as mapped key (before mapping)
system.autoMapOutlets = true;
system.mapClass(‘bClass’, B); // map again b/c B is changed
system.mapClass(‘c’, C);
// system.getObject(‘b’) would have c dep

65. COMMUNICATE IN BETWEEN
Now B has dependencies of A and C
How would you do in this scenario:
A is doing something
when it’s done, invoke a method of B
–this is a very common situation in player

66. MAP HANDLER AND NOTIFY
function A() {
this.say = function() {
system.notify(‘aDoneSaying’);
};
}
function B() {
this.a = undefined;
this.afterAsays = function(){
// do something
};
}
system.mapSingleton(‘a’,A); system.mapSingleton(‘b’, B);
system.mapHandler(‘aDoneSaying’, ’b’,‘afterAsays’);
// system.getObject(‘b’).a.say() would invoke b.afterAsays

67. NOTIFYING CLASS
system.mapSingleton(‘a’,A);
system.mapClass(‘b’, B); // map class here
var b1 = system.getObject(‘b’);
// b1.say() would invoke a newly instantiated b.afterAsays
// instead of b1.afterAsays
system.mapValue(‘b1’, b1);
system.unmapHandler(‘aDoneSaying’,‘b’,‘afterAsays’)
system.mapHandler(‘aDoneSaying’,‘b1’,‘afterAsays’)
// b1.say() would invoke b1.afterAsays

68. function E() {
this.a = undefined;
this.setup = function () {};
}
var e = new E();
system.injectInto(e);
// e.setup invoked
CONVENTION
setup method of module would be invoked
after getObject or injectInto

69. INJECT SYSTEM ITSELF
system.mapValue('system', system);
function D() {
this.system = undefined;
}
system.mapSingleton('d', D);
// system.getObject(‘d’) has system as dependency

70. function A() {}; function B() {}
function Dep() {
return {
system: undefined,
setup: function () {
this.system.autoMapOutlets = true;
// map dep here
this.system.mapSingleton('a',A);
this.system.mapClass('b', B);
}
};
}
function App() {
var b;
var system = new dijon.System();
system.mapValue('system', system);
system.mapOutlet('system');
var dep = new Dep();
system.injectInto(dep); // inject system to dep and invoke setup function, which map all deps
return {
system: undefined,
a: undefined,
setup: function () {
b = system.getObject('b');
},
init: function () {
system.injectInto(this); // after init, app.a exists
}
};
}
// after new App().init(), app get all dep setup in Dep

71. SUMMARY OF DIJONJS
• 3kb
• very little convention
• focus on di and notify / handler

72. BUILD PROJECT
• list all source files in index.html of testing page
• grunt as task manager
• concat and minify scripts with grunt-usemin
• export to dist/player.min.js

73. ONE LAST NOTICE
• ui should be totally separated
• ui <-> player -> videoElement
• ui order player by api, respond to player behavior
by event
player
videoElement
ui

74. FUTURE OFYAPI.JS
• firefox / safari drm support
• live
• ads
• smart tv / chromecast / nexus player

75. THANKYOU
questions ?