Download as PDF, PPTX
![{
"providers":
["elastictranscoder",
"elementalconductor",
"encodingcom",
"zencoder"],
"preset":
{
"name":
"sample_preset",
"description":
"This
is
an
example
preset",
"container":
"mp4",
"rateControl":
"VBR",
"video":
{
"profile":
"Main",
"profileLevel":
"3.1",
"height":
"720",
"width":
"1080",
"codec":
"h264",
"bitrate":
"1000000",
"gopSize":
"90",
"gopMode":
"fixed",
"interlaceMode":
"progressive"
},
"audio":
{
"codec":
"aac",
"bitrate":
"64000"
}
}
transcoding api - presets](https://image.slidesharecdn.com/presentation-170120111337/85/Snickers-Open-Source-HTTP-API-for-Media-Encoding-10-320.jpg)
![transcoding api - jobs
{
"provider":
"elastictranscoder",
"source":
“ftp://nytimes:isg00d@ftp.nytimes.com/folder/my_video_source.mov”,
“destination”:
“ftp://nytimes:isg00d@ftp.nytimes.com/outputs”,
"outputs":
[
{"preset":
"720p_mp4",
"fileName":
"my_video_720p.mp4"},
{"preset":
"1080p_mp4",
"fileName":
"my_video_1080p.mp4"},
{"preset":
"256p_hls",
"fileName":
"hls/my_video_480p.m3u8"},
{"preset":
"480p_hls",
"fileName":
"hls/my_video_480p.m3u8"},
{"preset":
"720p_hls",
"fileName":
"hls/my_video_720p.m3u8"},
{"preset":
"1080p_hls",
"fileName":
"hls/my_video_1080p.m3u8"},
{"preset":
"2160p_hls",
"fileName":
"hls/my_video_2160p.m3u8"}
],
"streamingParams":
{
"segmentDuration":
5,
"protocol":
"hls"
}
}](https://image.slidesharecdn.com/presentation-170120111337/85/Snickers-Open-Source-HTTP-API-for-Media-Encoding-11-320.jpg)
![var
Routes
=
map[Route]RouterArguments{
//Job
routes
CreateJob:
RouterArguments{Path:
"/jobs",
Method:
http.MethodPost},
ListJobs:
RouterArguments{Path:
"/jobs",
Method:
http.MethodGet},
GetJobDetails:
RouterArguments{Path:
"/jobs/{jobID}",
Method:
http.MethodGet},
StartJob:
RouterArguments{Path:
"/jobs/{jobID}/start",
Method:
http.MethodPost},
//Preset
routes
CreatePreset:
RouterArguments{Path:
"/presets",
Method:
http.MethodPost},
UpdatePreset:
RouterArguments{Path:
"/presets",
Method:
http.MethodPut},
ListPresets:
RouterArguments{Path:
"/presets",
Method:
http.MethodGet},
GetPresetDetails:
RouterArguments{Path:
"/presets/{presetName}",
Method:
http.MethodGet},
DeletePreset:
RouterArguments{Path:
"/presets/{presetName}",
Method:
http.MethodDelete},
}
how it works?
server/routes.go](https://image.slidesharecdn.com/presentation-170120111337/85/Snickers-Open-Source-HTTP-API-for-Media-Encoding-21-320.jpg)
Uploaded byFlávio Ribeiro
Snickers: Open Source HTTP API for Media Encoding
The document discusses an open-source HTTP API for media encoding developed by a senior engineer from The New York Times. It describes the system's architecture, including its microservices for fast encoding, scalability, and reliability, as well as details about the transcoding API and its operation. The future plans include multibitrate HLS support, a Go client, and the addition of another encoding engine.
More Related Content
Similar to Snickers: Open Source HTTP API for Media Encoding
Snickers: Open Source HTTP API for Media Encoding
- 1. Open Source HTTPAPI for Media Encoding Snickers GoLab 2017. Florence, Italy
- 2. /flavioribeiro /flavioribeiro flavioribeiro.com senior engineer @the new york times
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10. { "providers": ["elastictranscoder", "elementalconductor", "encodingcom", "zencoder"], "preset": { "name": "sample_preset", "description": "This is an example preset", "container": "mp4", "rateControl": "VBR", "video": { "profile": "Main", "profileLevel": "3.1", "height": "720", "width": "1080", "codec": "h264", "bitrate": "1000000", "gopSize": "90", "gopMode": "fixed", "interlaceMode": "progressive" }, "audio": { "codec": "aac", "bitrate": "64000" } } transcoding api - presets
- 11. transcoding api -jobs { "provider": "elastictranscoder", "source": “ftp://nytimes:isg00d@ftp.nytimes.com/folder/my_video_source.mov”, “destination”: “ftp://nytimes:isg00d@ftp.nytimes.com/outputs”, "outputs": [ {"preset": "720p_mp4", "fileName": "my_video_720p.mp4"}, {"preset": "1080p_mp4", "fileName": "my_video_1080p.mp4"}, {"preset": "256p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "480p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "720p_hls", "fileName": "hls/my_video_720p.m3u8"}, {"preset": "1080p_hls", "fileName": "hls/my_video_1080p.m3u8"}, {"preset": "2160p_hls", "fileName": "hls/my_video_2160p.m3u8"} ], "streamingParams": { "segmentDuration": 5, "protocol": "hls" } }
- 12. • old systemworked well for ~10 years, not anymore • new system is a set of microservices in Go • fast encoding, scalability, reliability • transcoding API is a wrapper for encoding services • presets, encoding jobs context recap
- 13.
- 14. motivation • team fluentin Go — except me
- 15. motivation • team fluentin Go — except me • I wanted to test different approaches
- 16. motivation • team fluentin Go — except me • I wanted to test different approaches • What if I create another encoding service in Go?
- 17. • open sourcealternative to encoding providers • deploy & run everywhere • add features any time • compatible with transcoding api jobs and presets What if I create another encoding service in Go?
- 19.
- 20. RESTful API how itworks?
- 21. var Routes = map[Route]RouterArguments{ //Job routes CreateJob: RouterArguments{Path: "/jobs", Method: http.MethodPost}, ListJobs: RouterArguments{Path: "/jobs", Method: http.MethodGet}, GetJobDetails: RouterArguments{Path: "/jobs/{jobID}", Method: http.MethodGet}, StartJob: RouterArguments{Path: "/jobs/{jobID}/start", Method: http.MethodPost}, //Preset routes CreatePreset: RouterArguments{Path: "/presets", Method: http.MethodPost}, UpdatePreset: RouterArguments{Path: "/presets", Method: http.MethodPut}, ListPresets: RouterArguments{Path: "/presets", Method: http.MethodGet}, GetPresetDetails: RouterArguments{Path: "/presets/{presetName}", Method: http.MethodGet}, DeletePreset: RouterArguments{Path: "/presets/{presetName}", Method: http.MethodDelete}, } how it works? server/routes.go
- 22.
- 23. RESTful API database pipeline downloaders uploadersencoding engines how it works?
- 24. func StartJob(cfg gonfig.Gonfig, dbInstance db.Storage, job types.Job) { newJob, err := SetupJob(job.ID, dbInstance, cfg) job = *newJob if err != nil { updateJobWithError(dbInstance, job, err.Error()) return } downloadFunc := downloaders.GetDownloadFunc(job.Source) if err := downloadFunc(log, cfg, dbInstance, job.ID); err != nil { updateJobWithError(dbInstance, job, err.Error()) return } encodeFunc := encoders.GetEncodeFunc(job) if err := encodeFunc(logger, dbInstance, job.ID); err != nil { updateJobWithError(dbInstance, job, err.Error()) return } uploadFunc := uploaders.GetUploadFunc(job.Destination) if err := uploadFunc(logger, dbInstance, job.ID); err != nil { updateJobWithError(dbInstance, job, err.Error()) return } CleanSwap(dbInstance, job.ID); err != nil job.Status = types.JobFinished dbInstance.UpdateJob(job.ID, job) } how it works? pipeline/pipeline.go
- 25. RESTful API database pipeline downloaders uploadersencoding engines ffmpeg binding how it works?
- 26. how it works? •encoding engines • Cgo wrapper for FFmpeg functions • https://github.com/3d0c/gmf
- 27. how it works? package segmenter /* #include <stdio.h> #include "libavformat/avformat.h" #include <libavdevice/avdevice.h> #include "c/segmenter.h" #include "c/util.h" #cgo LDFLAGS: -‐L${SRCDIR}/../build -‐lsegmenter -‐lavcodec -‐lavformat -‐lavutil */ import "C" import ( "fmt" "os" "unsafe" "github.com/3d0c/gmf" )
- 28.
- 30.
- 31.
- 32. future • multibitrate HLS •go client (WIP) • add another encoding engine using GStreamer
- 33. We’re hiring nyti.ms/technology @NYTDevs |developers.nytimes.com
- 34. Stay updated open.blogs.nytimes.com @NYTDevs |developers.nytimes.com
- 35. Connect with uson Slack! http://video-dev.org @NYTDevs | developers.nytimes.com
- 36. thank you! { ,} /flavioribeiro