Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building a Video Encoding Pipeline at The New York Times

These slides were presented on the Streaming Media West conference in 2016. This talk is also a reference for the blog post "Using Microservices to Encode and Publish Videos at The New York Times" at The New York Times Open blog.

- Streaming Media West 2016: http://streamingmedia.com/Conferences/West2016/
- Open Blog: http://open.blogs.nytimes.com/2016/11/01/using-microservices-to-encode-and-publish-videos-at-the-new-york-times/

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

Building a Video Encoding Pipeline at The New York Times

  1. 1. Building a Video Encoding Pipeline at The New York Times Streaming Media West, 2016
  2. 2. Maxwell Dayvson Da Silva Sr. Director of Engineering http://github.com/dayvson http://twitter.com/dayvson http://dayvson.me
  3. 3. Flávio Ribeiro Sr. Software Engineer http://github.com/flavioribeiro http://twitter.com/flavioribeiro http://flavioribeiro.com
  4. 4. First, a little bit of history
  5. 5. The Architecture of the old system Video CMS (StoryLog) NYTimes CMS (Scoop) Internal Encoding Cluster Internal Storage CDN
  6. 6. The flaws
  7. 7. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMSs are required to publish a video.
  8. 8. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4
  9. 9. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Production ID
  10. 10. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Version
  11. 11. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Slug
  12. 12. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Resolution
  13. 13. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video
  14. 14. The Flaws 1. Filenames 2. Several watch folders 3. U 4. Hard to scale (cluster in-house)
  15. 15. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video
  16. 16. The Flaws
  17. 17. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video
  18. 18. Number of videos published per year.
  19. 19. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMSs to publish a video
  20. 20. The challenges
  21. 21. The Challenges 1. Fast encoding. News Clips are very time sensitive 2. Scalability 3. Reliability 4. Company-wide video transcoding solution 5. High-quality transcoding 360°, Drone footage, Live Action
  22. 22. The solution
  23. 23. The Video Pipeline Distribution API Acquisition API Transcoding API Storage Media Factory Client Media Factory API CDN Database
  24. 24. Media Factory Client Distribution API Acquisition API Transcoding API Storage Media Factory Client Media Factory API Database CDN
  25. 25. Media Factory Client 1. EcmaScript 6 Library 2. Upload Files from Browser to S3 http://github.com/TTLabs/EvaporateJS 3. Trigger Jobs on Media Factory API 4. Used by other teams
  26. 26. Acquisition API Distribution API Acquisition API Transcoding API Storage Media Factory Client Database Media Factory API CDN
  27. 27. Acquisition API 1. Back-end integration to upload files External partners 2. Plan to support resumable uploads
  28. 28. Database Media Factory Client Storage Media Factory API Distribution API CDN Acquisition API Transcoding APIMedia Factory API
  29. 29. Media Factory API 1. All NYTimes business logics Interact with our internal APIs Define which presets will be used based on job type (Standard, 360) http://github.com/nytimes/video-presets
  30. 30. Transcoding API Distribution API Acquisition API Transcoding API Storage Media Factory Client Media Factory API CDN Database
  31. 31. Transcoding API 1. Integration with multiple encoding cloud services 2. Reliability & Scalability 3. Define a common Job specification 4. Define a common Preset specification
  32. 32. Transcoding API - Job Example { "provider": "elastictranscoder", "source": "s3://somebucket/folder/my_video.mov", "streamingParams": { "playlistFileName": "hls/playlist.m3u8", "segmentDuration": 5, "protocol": "hls" } "outputs": [ {"preset": "720p_mp4", "fileName": "my_video_720p.mp4"}, {"preset": "1080p_mp4", "fileName": "my_video_1080p.mp4"}, {"preset": "256p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "480p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "720p_hls", "fileName": "hls/my_video_720p.m3u8"}, {"preset": "1080p_hls", "fileName": "hls/my_video_1080p.m3u8"}, {"preset": "2160p_hls", "fileName": "hls/my_video_2160p.m3u8"} ] }
  33. 33. Transcoding API - Preset Example "providers": ["elastictranscoder", "elementalconductor", "encodingcom", "zencoder"], "preset": { "name": "sample_preset", "description": "This is an example preset", "container": "mp4", "profile": "Main", "profileLevel": "3.1", "rateControl": "VBR", "video": { "height": "720", "width": "1080", "codec": "h264", "bitrate": "1000000", "gopSize": "90", "gopMode": "fixed", "interlaceMode": "progressive" }, "audio": { "codec": "aac", "bitrate": "64000" }
  34. 34. Transcoding API 1. Integration with multiple encoding cloud services 2. Reliability & Scalability 3. Define a common Job specification 4. Define a common Preset specification 5. http://nyti.ms/mediafactory
  35. 35. The Distribution API Database Media Factory API Distribution API Acquisition API Media Factory Client Transcoding API Storage CDN
  36. 36. Distribution API 1. Fast file transfer leveraging Aspera protocol 2. Plan to support distribution to our partners Youtube, Facebook, other partners 3. Plan to support to multiple CDNs
  37. 37. Just a recap! Distribution API Acquisition API Transcoding API Storage Media Factory Client Media Factory API CDN Database
  38. 38. Future 1. Video Quality Classify into groups VMAF, Reingest on the Media Factory API 2. Open Source Distribution API Acquisition API 3. Snickers (http://github.com/snickers/snickers)
  39. 39. We’re hiring nyti.ms/technology @NYTDevs | developers.nytimes.com
  40. 40. Stay updated open.blogs.nytimes.com @NYTDevs | developers.nytimes.com
  41. 41. Connect with us on Slack! http://video-dev.org @NYTDevs | developers.nytimes.com
  42. 42. Thank you! { , } /flavioribeiro { , } /dayvson

×