Snickers, an open source alternative to encoding providers, presented at GoLab.io in Florence, Italy.

1. Open Source HTTP API for Media Encoding
Snickers
GoLab 2017. Florence, Italy

2. /flavioribeiro
/flavioribeiro
flavioribeiro.com
senior engineer @ the new york times

3. context

4. context
motivation

5. context
motivation
how it works?

6. context
motivation
how it works?
future

7. context

8. context
media factory
client
acquistion
API
media factory
API
transcoding
API
distribution
API
CDN
Storage
database
http://nyt.ms/mediafactory

9. transcoding api
transcoding
API
http://github.com/nytimes/video-transcoding-api

10. {
"providers":
["elastictranscoder",
"elementalconductor",
"encodingcom",
"zencoder"],
"preset":
{
"name":
"sample_preset",
"description":
"This
is
an
example
preset",
"container":
"mp4",
"rateControl":
"VBR",
"video":
{
"profile":
"Main",
"profileLevel":
"3.1",
"height":
"720",
"width":
"1080",
"codec":
"h264",
"bitrate":
"1000000",
"gopSize":
"90",
"gopMode":
"fixed",
"interlaceMode":
"progressive"
},
"audio":
{
"codec":
"aac",
"bitrate":
"64000"
}
}
transcoding api - presets

11. transcoding api - jobs
{
"provider":
"elastictranscoder",
"source":
“ftp://nytimes:isg00d@ftp.nytimes.com/folder/my_video_source.mov”,
“destination”:
“ftp://nytimes:isg00d@ftp.nytimes.com/outputs”,
"outputs":
[
{"preset":
"720p_mp4",
"fileName":
"my_video_720p.mp4"},
{"preset":
"1080p_mp4",
"fileName":
"my_video_1080p.mp4"},
{"preset":
"256p_hls",
"fileName":
"hls/my_video_480p.m3u8"},
{"preset":
"480p_hls",
"fileName":
"hls/my_video_480p.m3u8"},
{"preset":
"720p_hls",
"fileName":
"hls/my_video_720p.m3u8"},
{"preset":
"1080p_hls",
"fileName":
"hls/my_video_1080p.m3u8"},
{"preset":
"2160p_hls",
"fileName":
"hls/my_video_2160p.m3u8"}
],
"streamingParams":
{
"segmentDuration":
5,
"protocol":
"hls"
}
}

12. • old system worked well for ~10 years, not anymore
• new system is a set of microservices in Go
• fast encoding, scalability, reliability
• transcoding API is a wrapper for encoding services
• presets, encoding jobs
context recap

13. motivation

14. motivation
• team fluent in Go — except me

15. motivation
• team fluent in Go — except me
• I wanted to test different approaches

16. motivation
• team fluent in Go — except me
• I wanted to test different approaches
• What if I create another encoding service in Go?

17. • open source alternative to encoding providers
• deploy & run everywhere
• add features any time
• compatible with transcoding api jobs and presets
What if I create another encoding service in Go?

19. how it works?

20. RESTful API
how it works?

21. var
Routes
=
map[Route]RouterArguments{
//Job
routes
CreateJob:
RouterArguments{Path:
"/jobs",
Method:
http.MethodPost},
ListJobs:
RouterArguments{Path:
"/jobs",
Method:
http.MethodGet},
GetJobDetails:
RouterArguments{Path:
"/jobs/{jobID}",
Method:
http.MethodGet},
StartJob:
RouterArguments{Path:
"/jobs/{jobID}/start",
Method:
http.MethodPost},
//Preset
routes
CreatePreset:
RouterArguments{Path:
"/presets",
Method:
http.MethodPost},
UpdatePreset:
RouterArguments{Path:
"/presets",
Method:
http.MethodPut},
ListPresets:
RouterArguments{Path:
"/presets",
Method:
http.MethodGet},
GetPresetDetails:
RouterArguments{Path:
"/presets/{presetName}",
Method:
http.MethodGet},
DeletePreset:
RouterArguments{Path:
"/presets/{presetName}",
Method:
http.MethodDelete},
}
how it works?
server/routes.go

22. RESTful API
database
how it works?

23. RESTful API
database
pipeline
downloaders uploaders encoding engines
how it works?

24. func
StartJob(cfg
gonfig.Gonfig,
dbInstance
db.Storage,
job
types.Job)
{
newJob,
err
:=
SetupJob(job.ID,
dbInstance,
cfg)
job
=
*newJob
if
err
!=
nil
{
updateJobWithError(dbInstance,
job,
err.Error())
return
}
downloadFunc
:=
downloaders.GetDownloadFunc(job.Source)
if
err
:=
downloadFunc(log,
cfg,
dbInstance,
job.ID);
err
!=
nil
{
updateJobWithError(dbInstance,
job,
err.Error())
return
}
encodeFunc
:=
encoders.GetEncodeFunc(job)
if
err
:=
encodeFunc(logger,
dbInstance,
job.ID);
err
!=
nil
{
updateJobWithError(dbInstance,
job,
err.Error())
return
}
uploadFunc
:=
uploaders.GetUploadFunc(job.Destination)
if
err
:=
uploadFunc(logger,
dbInstance,
job.ID);
err
!=
nil
{
updateJobWithError(dbInstance,
job,
err.Error())
return
}
CleanSwap(dbInstance,
job.ID);
err
!=
nil
job.Status
=
types.JobFinished
dbInstance.UpdateJob(job.ID,
job)
}
how it works?
pipeline/pipeline.go

25. RESTful API
database
pipeline
downloaders uploaders encoding engines
ffmpeg binding
how it works?

26. how it works?
• encoding engines
• Cgo wrapper for FFmpeg functions
• https://github.com/3d0c/gmf

27. how it works?
package
segmenter
/*
#include
<stdio.h>
#include
"libavformat/avformat.h"
#include
<libavdevice/avdevice.h>
#include
"c/segmenter.h"
#include
"c/util.h"
#cgo
LDFLAGS:
-­‐L${SRCDIR}/../build
-­‐lsegmenter
-­‐lavcodec
-­‐lavformat
-­‐lavutil
*/
import
"C"
import
(
"fmt"
"os"
"unsafe"
"github.com/3d0c/gmf"
)

28. DEMO!

30. future

31. future
transcoding
API

32. future
• multibitrate HLS
• go client (WIP)
• add another encoding engine using GStreamer

33. We’re hiring
nyti.ms/technology
@NYTDevs | developers.nytimes.com

34. Stay updated
open.blogs.nytimes.com
@NYTDevs | developers.nytimes.com

35. Connect with us on Slack!
http://video-dev.org
@NYTDevs | developers.nytimes.com

36. thank you!
{ , } /flavioribeiro