What’s Apple planning for its media frameworks in the next 12 months? What’s it doing with Apple TV, or the HTTP Live Streaming standard? We won’t know until the curtain drops on WWDC! In this talk, we’ll amass everything audio- and video-related that gets announced throughout the week, combine it with the solid base of frameworks already present in the Apple platforms, and figure out from there what we’re going to be playing with in 2018.

1. Fall Premieres:
Media for Apple Platforms in
Late 2017
Chris Adamson • @invalidname
CocoaConf Next Door • June, 2017

2. Agenda
• HEVC / H.265
• Apple Music API
• Changes to Existing Frameworks
• HTTP Live Streaming
• Developments outside of Apple

3. HEVC / H.265

4. H.265
• "High Efficiency Video Codec"
• Approximately double the data compression (or
much better quality) of H.264 at same bitrate
• Expected use-case: 4K video streaming
• Work began in 2010, standard approved in 2013
• Versions 2, 3, and 4 approved since then

8. Standardized Mac / iOS support Major Uses
MPEG-1 1993 QuickTime 2.5 Video CD
MPEG-2 1996
QuickTime 6
(optional)
DVD, Digital TV
H.264 2003
QuickTime 7
(MacOS X 10.4),
iPhone OS 1
Blu-Ray, Digital TV,
Streaming
H.265 2013
macOS High
Sierra, iOS 11
???

9. Patent Encumbrances
• HEVC contains technology covered by hundreds
(thousands?) of patents
• Patent Pools: MPEG-LA, HEVC Advance, Velos
Media
• Don't even cover everything (Technicolor SA)
• Much costlier than H.264 licenses (max $25M/yr
[MPEG-LA] $40M/yr [HEVC Advance])

10. VP9
• Google-supported competitor to HEVC
• Royalty-free
• Only patents (🤞) are held by Google
• Supported by 75% of browsers (desktop +
mobile)
• Used extensively by YouTube for delivery to non-
Apple platforms

12. AV1
• Royalty-free codec from Alliance for Open Media
(AV1 means "AOMedia Video 1")
• Founding members: Amazon, Cisco, Intel,
Google, Microsoft, Mozilla, Netflix
• Successor to VP9
• Planned for release in 2017

13. http://www.streamingmediaglobal.com/Articles/ReadArticle.aspx?
ArticleID=118062
Then I asked where HEVC fit in. Aaron explained that Netflix currently
deployed HEVC primarily on Smart TVs, and saw the codec as integral to
its HDR strategy. For computers and mobile, however, H.264 and VP9 are
Netflix's primary codecs, and the focus of most of its current research,
which will soon include AV1.
I asked about the future of HEVC given the imminent release of AV1, and
the fact that Netflix was a founding member of the Alliance for Open
Media. Ronca laughed, explaining that Netflix was still encoding in VC1
format for some older platforms, and that HEVC is important for their UHD/
HDR experience, though he noted that the HEVC license uncertainty is an
ongoing concern.
NAB 17 Codec Roundup

14. Jan Ozer
HEVC will do well in broadcast, no doubt. Still not available in any
browser, iOS, and Netflix prefers VP9/AV1 over HEVC for Android. VP9
gets you most browsers and many smart TVs and OTT boxes (like Roku
4), so it's the smart money UHD codec if you don't need HDR.
http://www.streamingmediaglobal.com/Articles/
ReadArticle.aspx?ArticleID=118062

15. High Dynamic Range (HDR)
http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/HDR-
The-More-Bits-the-Better-118643.aspx
HDR: The More Bits, the Better
Apple's new Macs feature 10-bit HEVC decoding for HDR video viewing. To understand why that's
important, it's worth a look at an old technology—film cameras
By Tim Siglin
Posted on June 6, 2017
The aperture itself determines how deep or shallow the focus will be, which
is where we get the terminology for depth of field that's now being tossed
about for consumer smartphones. The Apple iPhone 7 "depth effect" feature
makes use of two lenses to create a digital version of depth of field.

16. HEVC in code
• AVVideoCodecType.hevc
• AVAssetExportPresetHEVC1920x1080, …3840x2160
• AVOutputSettingsPreset.hevc1920x1080,
hevc3840x2160
• kCMVideoCodecType_HEVC
• kVTCompressionPropertyKey_ProfileLevel values:
Profile0_Level10, Profile0_Level45, Profile3_Level45

18. Apple Music API

19. AppleMusic TODOs
• Web API (MusicKit) for getting Apple Music
artists/albums/playlists, getting user's
preferences
• Working with Store Kit to actually access those
items
• Playing with Media Player

20. MusicKit web service
• APIs: artists, albums, songs, playlists, search,
charts, recommendations
• Must have a developer token to call
• Must get a user token to access personalized
features
https://api.music.apple.com/{version}/catalog/{storefront}/{api}?[params]

21. Subscription UI
• If user isn't an Apple Music subscriber, you can
show the subscription view, optionally showing
specific content
• Can embed your affil token

22. Playback
• MPMediaPlayer
• Can create MPMediaItem (local songs),
MPMusicPlayerStoreQueueDescriptor (Apple
Music songs),
MPMusicPlayerPlayParametersQueueDescript
or (personalized collections)

24. Changes to Existing
Frameworks

25. AVAudioEngine
• Offline Rendering
• Auto shutdown
• Completion callbacks
(.dataConsumed, .dataRendered, .dataPlayedB
ack)
• Important: AUGraph (Audio Toolbox) to be
deprecated in 2018

28. Also
• watchOS recording
• AUAudioUnit "preferred view configuration" API
• Support for FLAC and Opus audio codecs
• Spatial audio formats

30. HTTP Live Streaming

31. HLS updates
• HLS approved for an actual RFC
• Format war with MPEG-DASH continues
• HEVC supported (of course), fMP4 only (not .ts)
• New subtitle standard ISMC1
• Nice but esoteric features (variables in playlists)
• FairPlay is still very much a thing. 🤐

33. Beyond Apple Park

34. ReplayKit
• Screen Capture + Livestreaming API

38. RTMP HLS

39. https://github.com/shogo4405/GPUHaishinKit.swift

40. https://github.com/shogo4405/GPUHaishinKit.swift

41. Fall Premieres:
Media for Apple Platforms in
Late 2017
Chris Adamson • @invalidname
CocoaConf Next Door • June, 2017