Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Building a Video
Encoding Pipeline at
The New York Times
Streaming Media West, 2016
Maxwell Dayvson Da Silva
Sr. Director of Engineering
http://github.com/dayvson
http://twitter.com/dayvson
http://dayvson.me
Flávio Ribeiro
Sr. Software Engineer
http://github.com/flavioribeiro
http://twitter.com/flavioribeiro
http://flavioribeiro...
First, a little bit of history
The Architecture of the old system
Video CMS
(StoryLog)
NYTimes CMS
(Scoop)
Internal
Encoding Cluster
Internal Storage
CDN
The flaws
The Flaws
1. The whole process relies on a filename
2. Several watch folders
3. Unstable transcoding
4. Hard to scale
5. T...
The Flaws
1. The whole process relies on a filename
2. Several watch folders (hard to maintain)
3. Unstable transcoding (p...
The Flaws
1. The whole process relies on a filename
2. Several watch folders (hard to maintain)
3. Unstable transcoding (p...
The Flaws
1. The whole process relies on a filename
2. Several watch folders (hard to maintain)
3. Unstable transcoding (p...
The Flaws
1. The whole process relies on a filename
2. Several watch folders (hard to maintain)
3. Unstable transcoding (p...
The Flaws
1. The whole process relies on a filename
2. Several watch folders (hard to maintain)
3. Unstable transcoding (p...
The Flaws
1. The whole process relies on a filename
2. Several watch folders
3. Unstable transcoding
4. Hard to scale
5. T...
The Flaws
1. Filenames
2. Several watch folders
3. U
4. Hard to scale (cluster in-house)
The Flaws
1. The whole process relies on a filename
2. Several watch folders
3. Unstable transcoding
4. Hard to scale
5. T...
The Flaws
The Flaws
1. The whole process relies on a filename
2. Several watch folders
3. Unstable transcoding
4. Hard to scale
5. T...
Number of videos
published per year.
The Flaws
1. The whole process relies on a filename
2. Several watch folders
3. Unstable transcoding
4. Hard to scale
5. T...
The challenges
The Challenges
1. Fast encoding. News Clips are very time sensitive
2. Scalability
3. Reliability
4. Company-wide video tr...
The solution
The Video Pipeline
Distribution API
Acquisition API
Transcoding API
Storage
Media Factory Client
Media Factory API
CDN
Dat...
Media Factory Client
Distribution API
Acquisition API
Transcoding API
Storage
Media Factory Client
Media Factory API
Datab...
Media Factory Client
1. EcmaScript 6 Library
2. Upload Files from Browser to S3
http://github.com/TTLabs/EvaporateJS
3. Tr...
Acquisition API
Distribution API
Acquisition API
Transcoding API
Storage
Media Factory Client
Database
Media Factory API
C...
Acquisition API
1. Back-end integration to upload files
External partners
2. Plan to support resumable uploads
Database
Media Factory Client
Storage
Media Factory API
Distribution API
CDN
Acquisition API
Transcoding APIMedia Factory ...
Media Factory API
1. All NYTimes business logics
Interact with our internal APIs
Define which presets will be used based o...
Transcoding API
Distribution API
Acquisition API
Transcoding API
Storage
Media Factory Client
Media Factory API
CDN
Databa...
Transcoding API
1. Integration with multiple encoding cloud
services
2. Reliability & Scalability
3. Define a common Job s...
Transcoding API - Job Example
{
"provider": "elastictranscoder",
"source": "s3://somebucket/folder/my_video.mov",
"streami...
Transcoding API - Preset Example
"providers": ["elastictranscoder", "elementalconductor", "encodingcom", "zencoder"],
"pre...
Transcoding API
1. Integration with multiple encoding cloud
services
2. Reliability & Scalability
3. Define a common Job s...
The Distribution API
Database
Media Factory API Distribution API
Acquisition API
Media Factory Client
Transcoding API
Stor...
Distribution API
1. Fast file transfer leveraging Aspera protocol
2. Plan to support distribution to our partners
Youtube,...
Just a recap!
Distribution API
Acquisition API
Transcoding API
Storage
Media Factory Client
Media Factory API
CDN
Database
Future
1. Video Quality
Classify into groups
VMAF, Reingest on the Media Factory API
2. Open Source
Distribution API
Acqui...
We’re hiring
nyti.ms/technology
@NYTDevs | developers.nytimes.com
Stay updated
open.blogs.nytimes.com
@NYTDevs | developers.nytimes.com
Connect with us on Slack!
http://video-dev.org
@NYTDevs | developers.nytimes.com
Thank you!
{ , } /flavioribeiro
{ , } /dayvson
Building a Video Encoding Pipeline at The New York Times
Upcoming SlideShare
Loading in …5
×

Building a Video Encoding Pipeline at The New York Times

These slides were presented on the Streaming Media West conference in 2016. This talk is also a reference for the blog post "Using Microservices to Encode and Publish Videos at The New York Times" at The New York Times Open blog.

- Streaming Media West 2016: http://streamingmedia.com/Conferences/West2016/
- Open Blog: http://open.blogs.nytimes.com/2016/11/01/using-microservices-to-encode-and-publish-videos-at-the-new-york-times/

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

Building a Video Encoding Pipeline at The New York Times

  1. 1. Building a Video Encoding Pipeline at The New York Times Streaming Media West, 2016
  2. 2. Maxwell Dayvson Da Silva Sr. Director of Engineering http://github.com/dayvson http://twitter.com/dayvson http://dayvson.me
  3. 3. Flávio Ribeiro Sr. Software Engineer http://github.com/flavioribeiro http://twitter.com/flavioribeiro http://flavioribeiro.com
  4. 4. First, a little bit of history
  5. 5. The Architecture of the old system Video CMS (StoryLog) NYTimes CMS (Scoop) Internal Encoding Cluster Internal Storage CDN
  6. 6. The flaws
  7. 7. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMSs are required to publish a video.
  8. 8. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4
  9. 9. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Production ID
  10. 10. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Version
  11. 11. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Slug
  12. 12. The Flaws 1. The whole process relies on a filename 2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Resolution
  13. 13. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video
  14. 14. The Flaws 1. Filenames 2. Several watch folders 3. U 4. Hard to scale (cluster in-house)
  15. 15. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video
  16. 16. The Flaws
  17. 17. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video
  18. 18. Number of videos published per year.
  19. 19. The Flaws 1. The whole process relies on a filename 2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMSs to publish a video
  20. 20. The challenges
  21. 21. The Challenges 1. Fast encoding. News Clips are very time sensitive 2. Scalability 3. Reliability 4. Company-wide video transcoding solution 5. High-quality transcoding 360°, Drone footage, Live Action
  22. 22. The solution
  23. 23. The Video Pipeline Distribution API Acquisition API Transcoding API Storage Media Factory Client Media Factory API CDN Database
  24. 24. Media Factory Client Distribution API Acquisition API Transcoding API Storage Media Factory Client Media Factory API Database CDN
  25. 25. Media Factory Client 1. EcmaScript 6 Library 2. Upload Files from Browser to S3 http://github.com/TTLabs/EvaporateJS 3. Trigger Jobs on Media Factory API 4. Used by other teams
  26. 26. Acquisition API Distribution API Acquisition API Transcoding API Storage Media Factory Client Database Media Factory API CDN
  27. 27. Acquisition API 1. Back-end integration to upload files External partners 2. Plan to support resumable uploads
  28. 28. Database Media Factory Client Storage Media Factory API Distribution API CDN Acquisition API Transcoding APIMedia Factory API
  29. 29. Media Factory API 1. All NYTimes business logics Interact with our internal APIs Define which presets will be used based on job type (Standard, 360) http://github.com/nytimes/video-presets
  30. 30. Transcoding API Distribution API Acquisition API Transcoding API Storage Media Factory Client Media Factory API CDN Database
  31. 31. Transcoding API 1. Integration with multiple encoding cloud services 2. Reliability & Scalability 3. Define a common Job specification 4. Define a common Preset specification
  32. 32. Transcoding API - Job Example { "provider": "elastictranscoder", "source": "s3://somebucket/folder/my_video.mov", "streamingParams": { "playlistFileName": "hls/playlist.m3u8", "segmentDuration": 5, "protocol": "hls" } "outputs": [ {"preset": "720p_mp4", "fileName": "my_video_720p.mp4"}, {"preset": "1080p_mp4", "fileName": "my_video_1080p.mp4"}, {"preset": "256p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "480p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "720p_hls", "fileName": "hls/my_video_720p.m3u8"}, {"preset": "1080p_hls", "fileName": "hls/my_video_1080p.m3u8"}, {"preset": "2160p_hls", "fileName": "hls/my_video_2160p.m3u8"} ] }
  33. 33. Transcoding API - Preset Example "providers": ["elastictranscoder", "elementalconductor", "encodingcom", "zencoder"], "preset": { "name": "sample_preset", "description": "This is an example preset", "container": "mp4", "profile": "Main", "profileLevel": "3.1", "rateControl": "VBR", "video": { "height": "720", "width": "1080", "codec": "h264", "bitrate": "1000000", "gopSize": "90", "gopMode": "fixed", "interlaceMode": "progressive" }, "audio": { "codec": "aac", "bitrate": "64000" }
  34. 34. Transcoding API 1. Integration with multiple encoding cloud services 2. Reliability & Scalability 3. Define a common Job specification 4. Define a common Preset specification 5. http://nyti.ms/mediafactory
  35. 35. The Distribution API Database Media Factory API Distribution API Acquisition API Media Factory Client Transcoding API Storage CDN
  36. 36. Distribution API 1. Fast file transfer leveraging Aspera protocol 2. Plan to support distribution to our partners Youtube, Facebook, other partners 3. Plan to support to multiple CDNs
  37. 37. Just a recap! Distribution API Acquisition API Transcoding API Storage Media Factory Client Media Factory API CDN Database
  38. 38. Future 1. Video Quality Classify into groups VMAF, Reingest on the Media Factory API 2. Open Source Distribution API Acquisition API 3. Snickers (http://github.com/snickers/snickers)
  39. 39. We’re hiring nyti.ms/technology @NYTDevs | developers.nytimes.com
  40. 40. Stay updated open.blogs.nytimes.com @NYTDevs | developers.nytimes.com
  41. 41. Connect with us on Slack! http://video-dev.org @NYTDevs | developers.nytimes.com
  42. 42. Thank you! { , } /flavioribeiro { , } /dayvson

×