Mobile Movies with HTTP Live Streaming (CocoaConf DC, March 2013)

1. Mobile Movies with HTTP Live Streaming Chris Adamson • @invalidname CocoaConf DC • March 23, 2013 • Herndon, VA Livestreaming at http://ustream.tv/channel/invalidstream Sides and code available on my blog: http://www.subfurther.com/blogMonday, March 25, 13

2. So Many Streaming Apps!Monday, March 25, 13

3. So Many Streaming Apps!Monday, March 25, 13

4. NetﬂixMonday, March 25, 13

5. FlixterMonday, March 25, 13

6. Network TV AppsMonday, March 25, 13

7. Network TV AppsMonday, March 25, 13

8. NBC Olympics Live ExtraMonday, March 25, 13

9. DirecTVMonday, March 25, 13

10. UStreamMonday, March 25, 13

11. CrunchyrollMonday, March 25, 13

12. TwitchTVMonday, March 25, 13

15. iTunes FestivalMonday, March 25, 13

16. Apple EventsMonday, March 25, 13

17. Oh, and it can be embedded in the <video> tagMonday, March 25, 13

18. Oh, and it can be embedded in the <video> tagMonday, March 25, 13

19. Sounds good? How do I get in on that?Monday, March 25, 13

20. What Youll Learn • What streaming is (and isnt) • Setting up HLS on the server • Using HLS streams in iOS apps • Real-world deploymentMonday, March 25, 13

21. HLS: What It Is (and isnt)Monday, March 25, 13

22. Good Ol BroadcastMonday, March 25, 13

23. Broadcast Media • Always live on some channel (a band of EM spectrum). • Every client tuned to that channel sees the same thing, at the same time. • One-way, one-to-many model.Monday, March 25, 13

24. Internet • Generally one-to-one (host to host). • Multicast IP is an exception, but is rare on the public Internet. • Two-way communication over sockets. • Routing can take many hops, via multiple transport media (wire, wiﬁ, cellular, etc.).Monday, March 25, 13

25. Impedence Mismatch!Monday, March 25, 13

26. Ye Olde Streaming • Client makes socket connection and keeps it open for duration of program. • Server sends media at playback speed (plus buffering). • Shoutcast: MP3 ﬁles served slowly over HTTP. • Typically use a special port number and special server software.Monday, March 25, 13

27. Streaming Problems • Difﬁcult and expensive to scale. • Special port numbers routinely blocked by businesses, ISPs, ﬁrewalls, etc. • Competing standards: Real Player, Windows Media, QuickTime (all with their own plugins). • No wonder Flash won. • Good luck holding a socket connection on cellular.Monday, March 25, 13

28. What If… • We didnt need an always-on socket connection? • We could just run over port 80? • We could just adopt industry standards like H.264 and AAC instead of cooking custom codecs?Monday, March 25, 13

29. HTTP Live Streaming • Serves media as a series of short ﬂat ﬁles, via HTTP, usually on port 80. • Any web server will do. • Client software reassembles the data into a continuous media stream. • Spec does not specify contents, but Apple uses H.264 and AAC, just like all their media apps.Monday, March 25, 13

30. Serving up HLS • Client URL is an .m3u8 playlist ﬁle • Playlist points to the media segment ﬁlesMonday, March 25, 13

31. Monday, March 25, 13

32. The HLS playlist #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-PLAYLIST-TYPE:VOD #EXTINF:9.975,! fileSequence0.ts #EXTINF:9.975,! fileSequence1.ts #EXTINF:9.975,! fileSequence2.ts #EXTINF:9.9767,! fileSequence3.ts #EXTINF:9.975,! [...] #EXT-X-ENDLISTMonday, March 25, 13

33. The HLS playlist #EXTM3U Format: .m3u8 format, #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 just a list of ﬁles to play #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-PLAYLIST-TYPE:VOD #EXTINF:9.975,! fileSequence0.ts #EXTINF:9.975,! fileSequence1.ts #EXTINF:9.975,! fileSequence2.ts #EXTINF:9.9767,! fileSequence3.ts #EXTINF:9.975,! [...] #EXT-X-ENDLISTMonday, March 25, 13

34. The HLS playlist #EXTM3U Format: .m3u8 format, #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 just a list of ﬁles to play #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-PLAYLIST-TYPE:VOD Metadata tags describe #EXTINF:9.975,! fileSequence0.ts the contents #EXTINF:9.975,! fileSequence1.ts #EXTINF:9.975,! fileSequence2.ts #EXTINF:9.9767,! fileSequence3.ts #EXTINF:9.975,! [...] #EXT-X-ENDLISTMonday, March 25, 13

35. The HLS playlist #EXTM3U Format: .m3u8 format, #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 just a list of ﬁles to play #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-PLAYLIST-TYPE:VOD Metadata tags describe #EXTINF:9.975,! fileSequence0.ts the contents #EXTINF:9.975,! fileSequence1.ts #EXTINF:9.975,! Each segment ﬁle fileSequence2.ts #EXTINF:9.9767,! preceded by metadata fileSequence3.ts (e.g., duration) #EXTINF:9.975,! [...] #EXT-X-ENDLISTMonday, March 25, 13

36. The HLS playlist #EXTM3U Format: .m3u8 format, #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 just a list of ﬁles to play #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-PLAYLIST-TYPE:VOD Metadata tags describe #EXTINF:9.975,! fileSequence0.ts the contents #EXTINF:9.975,! fileSequence1.ts #EXTINF:9.975,! Each segment ﬁle fileSequence2.ts #EXTINF:9.9767,! preceded by metadata fileSequence3.ts (e.g., duration) #EXTINF:9.975,! [...] If no end tag, client #EXT-X-ENDLIST refreshes periodicallyMonday, March 25, 13

37. <video src="prog_index.m3u8" width="640" height="480" controls autoplay>No video tag support</video>Monday, March 25, 13

38. How is this better than a ﬂat .m4v ﬁle? • Streams can provide variants for different bandwidths (as we’ll see…) • Segments make it easier to scrub into the video • Streams can be live videoMonday, March 25, 13

39. The “Live” in HLS • A playlist is a live stream if it doesn’t have an #EXT-X-ENDLIST tag • Live playlist will generally just contain the last minute or so of segments • Client will refresh playlist every minute or so, download whatever segments it doesn’t already have, queue them locally • “Live” isn’t really “live” (often a minute behind)Monday, March 25, 13

40. Serving up HLSMonday, March 25, 13

46. mediafilesegmenter • Splits an A/V file into segment files, creates the .m3u8 playlist • Source must be .mov or .m4v with H.264 video, AAC audio • Output segments will be MPEG-2 Transport Stream (.ts) files, or .aac if audio-only • Segment paths are relative, use -b to prepend URL stubMonday, March 25, 13

47. Technical Note TN2224 The following audio and video formats are supported: • Video: H.264 Baseline Profile Level 3.0 (iPhone/iPod Touch), Main Profile Level 3.1 (iPad 1,2) • Audio: HE-AAC or AAC-LC up to 48 kHz, stereo audio OR MP3 (MPEG-1 Audio Layer 3) 8 kHz to 48 kHz, stereo audio Note: iPhone 3G supports Baseline Profile Level 3.1. If your app runs on older iPhones, however, you should use H.264 Baseline Profile 3.0 for compatibility.Monday, March 25, 13

48. Yuna:HTTP Live Streaming tests cadamson$ mediafilesegmenter -f basic source/IMG_0251.MOV Jun 24 2012 10:01:24.203: Using floating point is not backward compatible to iOS 4.1 or earlier devices Jun 24 2012 10:01:24.204: Processing file /Users/cadamson/ Documents/HTTP Live Streaming tests/source/IMG_0251.MOV Jun 24 2012 10:01:24.338: Finalized /Users/cadamson/ Documents/HTTP Live Streaming tests/basic/fileSequence0.ts Jun 24 2012 10:01:24.375: segment bitrate 3.78028e+06 is new max Jun 24 2012 10:01:24.468: Finalized /Users/cadamson/ Documents/HTTP Live Streaming tests/basic/fileSequence1.ts Jun 24 2012 10:01:24.554: Finalized /Users/cadamson/ Documents/HTTP Live Streaming tests/basic/fileSequence2.ts Jun 24 2012 10:01:24.631: Finalized /Users/cadamson/ Documents/HTTP Live Streaming tests/basic/fileSequence3.ts Jun 24 2012 10:01:24.717: Finalized /Users/cadamson/ Documents/HTTP Live Streaming tests/basic/fileSequence4.tsMonday, March 25, 13

49. DemoMonday, March 25, 13

50. Variant Playlists • One bitrate does not fit all: Mac on Ethernet versus iPhone on Edge. • Solution: encode your video at multiple bitrates, offer metadata in playlist about whats available, let client figure out which to use. • HLS clients automatically switch to best variant for current network conditions, switch on the fly.Monday, March 25, 13

51. variantplaylistcreator • Creates a playlist that itself points to playlists created with mediafilesegmenter. • Each entry contains metadata describing the bitrate and encoding of the variant. • Tool takes argument pairs: file or URL of a variant .m3u8, and metadata .plist created with mediafilesegmenter -I flag • First entry in variant playlist is default; client will try this one firstMonday, March 25, 13

52. Encoding the variantsMonday, March 25, 13

53. TN2224 RecommendationsMonday, March 25, 13

54. Creating variants Yuna:HTTP Live Streaming tests cadamson$ mediafilesegmenter -I -f variants/ broadband/ source/IMG_0254_Broadband.m4v Yuna:HTTP Live Streaming tests cadamson$ mediafilesegmenter -I -f variants/wifi source/IMG_0254_WiFi.m4v Yuna:HTTP Live Streaming tests cadamson$ mediafilesegmenter -I -f variants/ cellular source/IMG_0254_Cellular.m4vMonday, March 25, 13

55. Creating variant playlist Yuna:HTTP Live Streaming tests cadamson$ variantplaylistcreator -o variants/ variants.m3u8 variants/broadband/ prog_index.m3u8 source/ IMG_0254_Broadband.plist variants/wifi/ prog_index.m3u8 source/ IMG_0254_WiFi.plist variants/cellular/ prog_index.m3u8 source/ IMG_0254_Cellular.plistMonday, March 25, 13

57. Thats Great, but… How do we keep people from stealing our stream?Monday, March 25, 13

58. Encryption • HLS encrypts files, not transport. • Easy to scale: still serving flat files, but now theyre useless without decryption keys. • Serving the keys still needs to be secure. • Necessary, but not sufficient, for DRM.Monday, March 25, 13

59. Encrypting a playlist Yuna:HTTP Live Streaming tests cadamson$ mediafilesegmenter - I -k keys -f encrypted/cellular source/IMG_0426_Cellular.m4v Jun 24 2012 18:59:47.115: Using new key/iv rotation period; this is not backward compatible to iOS 3.1.* or earlier devices. Use the "-encrypt-iv=sequence" option for compatibility with those devices. Jun 24 2012 18:59:47.115: Using floating point is not backward compatible to iOS 4.1 or earlier devices Jun 24 2012 18:59:47.115: Processing file /Users/cadamson/ Documents/HTTP Live Streaming tests/source/ IMG_0426_Cellular.m4v Jun 24 2012 18:59:47.152: changing IV Jun 24 2012 18:59:47.160: Finalized /Users/cadamson/ Documents/HTTP Live Streaming tests/encrypted/cellular/ fileSequence0.ts Jun 24 2012 18:59:47.160: segment bitrate 271257 is new maxMonday, March 25, 13

60. Encrypted .ts ﬁlesMonday, March 25, 13

64. Protect those keys /www/protected /www/hlsMonday, March 25, 13

66. Captions • HLS supports CEA-608 closed captions in the MPEG-2 Transport Stream • If using ﬁle segmenter, add a closed- caption track (type clcp) to your source QuickTime .mov • Or use Compressor and Sonic ScenaristMonday, March 25, 13

67. Captions in Netﬂix AppMonday, March 25, 13

68. Client-side HLSMonday, March 25, 13

69. Opening an HLS stream • Provide the .m3u8 URL to MPMoviePlayerController or AVPlayer • Add the movie view or layer to your UI, customizing size or scaling if necessaryMonday, March 25, 13

70. Create an MPMoviePlayerController // create new movie player self.moviePlayer = [[MPMoviePlayerController alloc] ! ! ! ! ! ! ! ! initWithContentURL:streamURL]; [self.moviePlayer prepareToPlay]; • This is the same as playing a local ﬁle or any other URLMonday, March 25, 13

71. Add it to your UI [self.moviePlayer.view setFrame: ! ! ! ! ! ! ! self.movieContainerView.bounds]; [self.movieContainerView addSubview: ! ! ! ! ! ! ! ! self.moviePlayer.view]; self.moviePlayer.scalingMode = ! ! ! ! ! ! ! ! MPMovieScalingModeFill; • Can inspect the moviePlayers naturalSize, though it may change during playback (listen for MPMovieNaturalSizeAvailableNotification), or just setFullscreen:animated:Monday, March 25, 13

72. MPMovieScalingMode AspectFitMonday, March 25, 13

73. MPMovieScalingMode AspectFillMonday, March 25, 13

74. MPMovieScalingMode FillMonday, March 25, 13

75. Accessing Encrypted Streams • Media Player and AV Foundation can use NSURLCredentials that youve provided • Place credentials in NSURLCredentialStorage • Server can provide the keys securely(*) with HTTP Basic or Digest authentication, an HTTPS script, etc. * - For various values of "secure"Monday, March 25, 13

76. Setting credentials (1) NSURLProtectionSpace *protectionSpace = ! [[NSURLProtectionSpace alloc] ! ! initWithHost:host ! ! port:port ! ! protocol:protocol ! ! realm:realm ! ! authenticationMethod: ! ! ! NSURLAuthenticationMethodDefault];Monday, March 25, 13

77. Setting credentials (2) NSURLCredential *credential = ! [NSURLCredential credentialWithUser:username ! ! ! ! ! ! ! ! ! password:password ! ! ! ! ! ! ! ! ! ! ! persistence: ! ! ! NSURLCredentialPersistenceForSession]; [[NSURLCredentialStorage ! sharedCredentialStorage] ! ! setDefaultCredential:credential ! ! forProtectionSpace:protectionSpace];Monday, March 25, 13

78. Live StreamingMonday, March 25, 13

79. Live Streaming • mediastreamsegmenter mostly works like the ﬁle version, but takes its input from UDP stream or a Unix pipe • Only difference is that .m3u8 ﬁle doesnt have an EXT-X-ENDLIST tag, so client reloads periodically to fetch new segments • How the heck do you create a UDP A/V stream?Monday, March 25, 13

80. You Dont • None of Apples tools create the required MPEG-2 stream • This is a "third party opportunity" • Which begs the question… buy or build?Monday, March 25, 13

81. Streaming in the Real World Its not all about iPhones…Monday, March 25, 13

82. Streaming Clients • Mobile Devices: iPhone, iPad, iPod Touch… plus Android, Windows Mobile, etc. • Mac and Windows PCs • Game consoles • Over-the-top (OTT) boxes: Apple TV, RokuMonday, March 25, 13

84. Playing HLS on Roku Roku "channels" are programmed in the "BrightScript" programming language: port = CreateObject("roMessagePort") screen = CreateObject("roVideoScreen") screen.SetMessagePort(port) screen.SetPositionNotificationPeriod(30) screen.SetContent(episode) screen.Show() episode is an "associative array" with key/value pairs for URLs, formats ("hls"), bitrates, etc.Monday, March 25, 13

85. Lets Get Practical • HLS is the preferred format for Roku • What other devices do you get for free? • What other devices do you have to be on? • How to encode and deliver to the devices you need?Monday, March 25, 13

86. HLS Alternatives • Flash still rules on the desktop/browser space, thanks in part to Mozillas obstinance about H.264 in <video> (irony alert: H.264 is the de facto standard for Flash video) • Adobe Dynamic Streaming and Microsoft Smooth Streaming are highly similar to HLS: bitrate-adaptive streams over HTTPMonday, March 25, 13

87. MPEG-DASH • Attempt at a standardized approach to HTTP adaptive-bitrate streaming. ISO/IEC 23009-1. http://xkcd.com/927/Monday, March 25, 13

88. Emerging Consensus • Flash for PCs • HTTP Live Streaming for iOS • Plus whatever other devices you need / are able to supportMonday, March 25, 13

89. Real-World HLS • Can you competently encode all your media at all the variant bitrates you need? • Do you have a way to QC all your streams? • Can you handle the server load?Monday, March 25, 13

90. Build or Buy: Services • Provide hosting, live transcoding, bandwidth • All-in-one: UStream, LiveStream, Justin.tv / Twitch.tv (all of which have iOS apps) • May provide broadcast tools (Flash applet, Telestream Wirecast, etc. • Often free with ads; you pay to go ad-free, embed on your site, etc.Monday, March 25, 13

91. invalidstream http://www.ustream.tv/channel/invalidstreamMonday, March 25, 13

92. Production Demo http://www.telestream.net/wirecast/Monday, March 25, 13

93. Content Delivery Networks http://en.wikipedia.org/wiki/Content_delivery_networkMonday, March 25, 13

94. Content Delivery Networks • CDNs host your media on edge servers that are closer to your clients. Less strain on your servers and the backbones. • Examples: Akamai, Limelight, EdgeCast • Big media companies may have their own CDN • Most already know how to do HLSMonday, March 25, 13

95. Buy or Build: Encoders $25,000 $4,000Monday, March 25, 13

96. Buy or Build: Bandwidth • Each HLS client will consume up to 1GB / hour, depending on variant bitrates, client bandwidth, etc. • Many CDNs charge around $0.20/GB.Monday, March 25, 13

97. Bandwidth Costs Prices from ScaleEngine, UStream, and LiveStream as of October 2012Monday, March 25, 13

101. Self-Hosted Costs Prices from MacMiniColo and Amazon EC2 as of March 2013Monday, March 25, 13

104. Wrap UpMonday, March 25, 13

105. Takeaways • HLS is a very practical streaming solution • Only part of the picture if youre multi-platform • Encoding and serving correctly requires some care and expertise, and a lot of money • Client-side software requirements are fairly simpleMonday, March 25, 13

106. Q&A Slides and code will be available on my blog: http://www.subfurther.com/blog http://www.slideshare.net/invalidname @invalidnameMonday, March 25, 13