Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Open Source HTTP API for Media Encoding
Snickers
GoLab 2017. Florence, Italy
/flavioribeiro
/flavioribeiro
flavioribeiro.com
senior engineer @ the new york times
context
context
motivation
context
motivation
how it works?
context
motivation
how it works?
future
context
context
media factory
client
acquistion
API
media factory
API
transcoding
API
distribution
API
CDN
Storage
database
http:/...
transcoding api
transcoding
API
http://github.com/nytimes/video-transcoding-api
{	
  
	
  	
  "providers":	
  ["elastictranscoder",	
  "elementalconductor",	
  "encodingcom",	
  "zencoder"],

	
  	
  "p...
transcoding api - jobs
{	
  
	
  	
  "provider":	
  "elastictranscoder",

	
  	
  "source":	
  “ftp://nytimes:isg00d@ftp.n...
• old system worked well for ~10 years, not anymore
• new system is a set of microservices in Go
• fast encoding, scalabil...
motivation
motivation
• team fluent in Go — except me
motivation
• team fluent in Go — except me
• I wanted to test different approaches
motivation
• team fluent in Go — except me
• I wanted to test different approaches
• What if I create another encoding serv...
• open source alternative to encoding providers
• deploy & run everywhere
• add features any time
• compatible with transc...
how it works?
RESTful API
how it works?
var	
  Routes	
  =	
  map[Route]RouterArguments{	
  
	
  	
  //Job	
  routes	
  
	
  	
  CreateJob:	
  	
  	
  	
  	
  Rou...
RESTful API
database
how it works?
RESTful API
database
pipeline
downloaders uploaders encoding engines
how it works?
func	
  StartJob(cfg	
  gonfig.Gonfig,	
  dbInstance	
  db.Storage,	
  job	
  types.Job)	
  {	
  
	
  	
  newJob,	
  err	
...
RESTful API
database
pipeline
downloaders uploaders encoding engines
ffmpeg binding
how it works?
how it works?
• encoding engines
• Cgo wrapper for FFmpeg functions
• https://github.com/3d0c/gmf
how it works?
package	
  segmenter	
  
/*	
  
#include	
  <stdio.h>	
  
#include	
  "libavformat/avformat.h"	
  
#include	...
DEMO!
future
future
transcoding
API
future
• multibitrate HLS
• go client (WIP)
• add another encoding engine using GStreamer
We’re hiring
nyti.ms/technology
@NYTDevs | developers.nytimes.com
Stay updated
open.blogs.nytimes.com
@NYTDevs | developers.nytimes.com
Connect with us on Slack!
http://video-dev.org
@NYTDevs | developers.nytimes.com
thank you!
{ , } /flavioribeiro
Snickers: Open Source HTTP API for Media Encoding
Snickers: Open Source HTTP API for Media Encoding
Upcoming SlideShare
Loading in …5
×

Snickers: Open Source HTTP API for Media Encoding

Snickers, an open source alternative to encoding providers, presented at GoLab.io in Florence, Italy.

  • Be the first to comment

  • Be the first to like this

Snickers: Open Source HTTP API for Media Encoding

  1. 1. Open Source HTTP API for Media Encoding Snickers GoLab 2017. Florence, Italy
  2. 2. /flavioribeiro /flavioribeiro flavioribeiro.com senior engineer @ the new york times
  3. 3. context
  4. 4. context motivation
  5. 5. context motivation how it works?
  6. 6. context motivation how it works? future
  7. 7. context
  8. 8. context media factory client acquistion API media factory API transcoding API distribution API CDN Storage database http://nyt.ms/mediafactory
  9. 9. transcoding api transcoding API http://github.com/nytimes/video-transcoding-api
  10. 10. {      "providers":  ["elastictranscoder",  "elementalconductor",  "encodingcom",  "zencoder"],
    "preset":  {
        "name":  "sample_preset",
        "description":  "This  is  an  example  preset",
        "container":  "mp4",
        "rateControl":  "VBR",
        "video":  {                  "profile":  "Main",  "profileLevel":  "3.1",
                "height":  "720",  "width":  "1080",
                "codec":  "h264",  "bitrate":  "1000000",
                "gopSize":  "90",  "gopMode":  "fixed",
                "interlaceMode":  "progressive"
        },
        "audio":  {
                "codec":  "aac",
                "bitrate":  "64000"
        }
 } transcoding api - presets
  11. 11. transcoding api - jobs {      "provider":  "elastictranscoder",
    "source":  “ftp://nytimes:isg00d@ftp.nytimes.com/folder/my_video_source.mov”,      “destination”:  “ftp://nytimes:isg00d@ftp.nytimes.com/outputs”,
    "outputs":  [
        {"preset":  "720p_mp4",  "fileName":  "my_video_720p.mp4"},
        {"preset":  "1080p_mp4",  "fileName":  "my_video_1080p.mp4"},
        {"preset":  "256p_hls",  "fileName":  "hls/my_video_480p.m3u8"},
        {"preset":  "480p_hls",  "fileName":  "hls/my_video_480p.m3u8"},
        {"preset":  "720p_hls",  "fileName":  "hls/my_video_720p.m3u8"},
        {"preset":  "1080p_hls",  "fileName":  "hls/my_video_1080p.m3u8"},
        {"preset":  "2160p_hls",  "fileName":  "hls/my_video_2160p.m3u8"}
    ],      "streamingParams":  {
        "segmentDuration":  5,
        "protocol":  "hls"
    }
 }  
  12. 12. • old system worked well for ~10 years, not anymore • new system is a set of microservices in Go • fast encoding, scalability, reliability • transcoding API is a wrapper for encoding services • presets, encoding jobs context recap
  13. 13. motivation
  14. 14. motivation • team fluent in Go — except me
  15. 15. motivation • team fluent in Go — except me • I wanted to test different approaches
  16. 16. motivation • team fluent in Go — except me • I wanted to test different approaches • What if I create another encoding service in Go?
  17. 17. • open source alternative to encoding providers • deploy & run everywhere • add features any time • compatible with transcoding api jobs and presets What if I create another encoding service in Go?
  18. 18. how it works?
  19. 19. RESTful API how it works?
  20. 20. var  Routes  =  map[Route]RouterArguments{      //Job  routes      CreateJob:          RouterArguments{Path:  "/jobs",  Method:  http.MethodPost},      ListJobs:            RouterArguments{Path:  "/jobs",  Method:  http.MethodGet},      GetJobDetails:  RouterArguments{Path:  "/jobs/{jobID}",  Method:  http.MethodGet},      StartJob:            RouterArguments{Path:  "/jobs/{jobID}/start",  Method:  http.MethodPost},      //Preset  routes      CreatePreset:          RouterArguments{Path:  "/presets",  Method:  http.MethodPost},      UpdatePreset:          RouterArguments{Path:  "/presets",  Method:  http.MethodPut},      ListPresets:            RouterArguments{Path:  "/presets",  Method:  http.MethodGet},      GetPresetDetails:  RouterArguments{Path:  "/presets/{presetName}",  Method:  http.MethodGet},      DeletePreset:          RouterArguments{Path:  "/presets/{presetName}",  Method:  http.MethodDelete},   } how it works? server/routes.go
  21. 21. RESTful API database how it works?
  22. 22. RESTful API database pipeline downloaders uploaders encoding engines how it works?
  23. 23. func  StartJob(cfg  gonfig.Gonfig,  dbInstance  db.Storage,  job  types.Job)  {      newJob,  err  :=  SetupJob(job.ID,  dbInstance,  cfg)      job  =  *newJob      if  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return    }      downloadFunc  :=  downloaders.GetDownloadFunc(job.Source)      if  err  :=  downloadFunc(log,  cfg,  dbInstance,  job.ID);  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return    }      encodeFunc  :=  encoders.GetEncodeFunc(job)      if  err  :=  encodeFunc(logger,  dbInstance,  job.ID);  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return      }      uploadFunc  :=  uploaders.GetUploadFunc(job.Destination)      if  err  :=  uploadFunc(logger,  dbInstance,  job.ID);  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return      }      CleanSwap(dbInstance,  job.ID);  err  !=  nil      job.Status  =  types.JobFinished      dbInstance.UpdateJob(job.ID,  job)   }   how it works? pipeline/pipeline.go
  24. 24. RESTful API database pipeline downloaders uploaders encoding engines ffmpeg binding how it works?
  25. 25. how it works? • encoding engines • Cgo wrapper for FFmpeg functions • https://github.com/3d0c/gmf
  26. 26. how it works? package  segmenter   /*   #include  <stdio.h>   #include  "libavformat/avformat.h"   #include  <libavdevice/avdevice.h>   #include  "c/segmenter.h"   #include  "c/util.h"   #cgo  LDFLAGS:  -­‐L${SRCDIR}/../build  -­‐lsegmenter  -­‐lavcodec  -­‐lavformat  -­‐lavutil   */   import  "C"   import  (     "fmt"     "os"     "unsafe"     "github.com/3d0c/gmf"   )  
  27. 27. DEMO!
  28. 28. future
  29. 29. future transcoding API
  30. 30. future • multibitrate HLS • go client (WIP) • add another encoding engine using GStreamer
  31. 31. We’re hiring nyti.ms/technology @NYTDevs | developers.nytimes.com
  32. 32. Stay updated open.blogs.nytimes.com @NYTDevs | developers.nytimes.com
  33. 33. Connect with us on Slack! http://video-dev.org @NYTDevs | developers.nytimes.com
  34. 34. thank you! { , } /flavioribeiro

×