Inside the ABC's new Media Transcoding system, Metro

metro - Media Transcoder
metro

Daphne Chong
Nick McHardy
@daphnechong
@nickmchardy

Questions?

Project Background
- What is Transcoding?
- Examples
- FFmpeg
- Why did we build our own?

Architecture
- Architecture overview
- Technology choices
- Transcoder autoscaling policies
- Costs

Future Roadmap
- Cost optimisations
- Validation and Introspection
- Features

Project Background:
What is
Transcoding?

Converting video
to different formats

High speed
Medium speed
Low speed

Standard Encoding

Project Background:
Transcoding
Example

Step 1: Prepare watermark

Step 2: Input video (GXF)

Step 3: Asymmetrical cropping

Step 4: Scale and set aspect ratio

Step 5: Apply watermark

Step 6: Ready to stream
1500k

other rendition sizes
1000k
650k
500k
220k
audio only

Original file audio track

Previous transcoder audio track

After audio normalisation

Transcoding profiles

GXF, ABC1
Profile:
- Crop
- Normalise audio
- Adjust ratio
- Add watermark
- GXF to MP4

GXF, ABC1

GXF, ABC1
MOV, Arts
Profile:
- Adjust ratio
- ProRes MOV to MP4

GXF, ABC1
MOV, Arts

Project Background:
FFmpeg

./ffmpeg utopia.gxf utopia.mp4

"hi+": {
"transcoder": "ffmpeg",
"transcoderoptions": "-c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,
scale=800x450,setdar=16/9[main], [1:v]scale=42:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-37:
main_h-overlay_h-34" -crf 22 -maxrate 904k -bufsize 1000k -pix_fmt yuv420p -profile:v high -level
4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2,
dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]"
-movflags +faststart -threads 0 -y",
"bitrate": "1000k",
"outputfilemimetype": "video/mp4"
},
"hi++": {
"transcoder": "ffmpeg",
"transcoderoptions": "-c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,
scale=1024x576,setdar=16/9[main], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:
main_h-overlay_h-44" -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -
level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1]
amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0
-map "[aout]" -movflags +faststart -threads 0 -y",
"bitrate": "1500k",
"outputfilemimetype": "video/mp4"
}
FFmpeg filters are fun

af_sidhaiompress f_senmd vfropdett vf_kerndeint vf_signalstats af_adelay af_sileedett f_zmq vfurves
vf_lenorrtion vf_smartblur af_aho af_sileeremove fifo vf_tdnoiz vf_libopev vf_spp af_aemphasis
af_sofalizer filtfmts vf_deband vf_lut vf_ssim af_afade af_stereotools formats vf_dimate vf_lut3d vf_stk
af_aformat af_stereowiden framepool vf_dejudder vf_maskedmerge vf_stereo3d af_agate af_tremolo
framesy vf_delogo vf_deint vf_subtitles af_alimiter af_vibrato generate_wave_table vf_deshake
vf_mergeplanes vf_super2xsai af_amerge af_volume graphdump vf_deteline vf_mpdimate vf_swapuv
af_amix af_volumedett graphparser vf_disple vf_neighbor vf_teline af_anequalizer allfilters lavfutils
vf_drawbox vf_noise vf_thumbnail af_anull asink_anullsink log2_tab vf_drawtext vf_null vf_tile af_apad
as_anoises lswsutils vf_edgedett vf_r vf_tinterle af_aphaser as_anulls opel_allkernels vf_elbg vf_overlay
vf_transpose af_apulsator as_flite pthread vf_eq vf_owdenoise vf_unsharp af_aresample as_sine setpts
vf_extrtplanes vf_pad vf_uspp af_asetnsamples audio settb vf_fade vf_palettegen vf_vtorope af_asetrate
avf_aphasemeter split vf_fftfilt vf_paletteuse vf_vflip af_ashowinfo avf_avtorope s_movie vf_field
vf_persptive vf_vidstabdett af_astats avfoat transform vf_fieldmah vf_phase vf_vidstabtransform af_asyts
avf_shoqt trim vf_fieldorder vf_pixdetest vf_vignette af_atempo avf_showfreqs unsharp_opel vf_find_rt
vf_pp vf_w3fdif af_biquads avf_showsptrum vf_alphamerge vf_format vf_pp7 vf_waveform af_bs2b
avf_showvolume vf_aspt vf_fps vf_psnr vf_xbr afhannelmap avf_showwaves vf_atadenoise vf_framepk
vf_pullup vf_yadif afhannelsplit avfilter vf_bbox vf_framerate vf_qp vf_zoompan afhorus avfiltergraph
vf_blkdett vf_framestep vf_random vf_zale afompand bbox vf_blkframe vf_frei0r vf_removegrain video
afompensationdelay buffersink vf_blend vf_fspp vf_removelogo vidstabutils af_shift buffers vf_boxblur
vf_geq vf_repeatfields vsink_nullsink af_dynaudnorm deshake_opel vfhromakey vf_gradfun vf_rotate
vsellauto af_earwax drawutils vfodview vf_hflip vf_sab vs_life af_extrastereo dualinput vfolorbalae
vf_histeq vf_ale vs_mandelbrot af_flanger f_drawgraph vfolohannelmixer vf_histogram vf_seltivolor
vs_mptests af_join f_ebur128 vfolorkey vf_hqdn3d vf_separatefields vs_tests af_ladspa f_interleave
vfolorlevels vf_hqx vf_setfield window_fu af_pan f_perms vfolormatrix vf_hue vf_showinfo af_replaygain
f_realtime vfopy vf_idet vf_showpalette af_resample f_reverse vfover_rt vf_il vf_shuffleframes
af_rubberband f_selt vfrop vf_interle vf_shuffleplanes
other FFmpeg filters

Project Background:
Why did we
build Metro?

thesupercarkids.com

mini-me.com

Thousands of
pieces of content

More customisation
& flexibility

Cost vs Speed
Scale
Flexibility

System Overview:
Architecture

“The Cloud”

8 minutes 27 seconds

Original
length
Time to produce rendition
60 min
7 min
42 min
Lowest
(audio)
Highest
(1500k)

System Overview:
Technology
Choices

Node + Go
- AWS: EC2, SQS,
- S3, Dynamo/RDS

Node.js
Components

Node.js
- Existing team experience
- Rapid development
- Solid choice for APIs
- Reservations for long lived processes
- Untyped. Sadface.

Golang
Components

Golang
- Typed and compiled
- Cross-platform
- Easy deployment
- Concurrency (we don’t use this heavily)

Golang
- Reasonable learning curve
- Magic interfaces
- Different techniques for testing
- Dependencies are challenging

AWS - EC2
Elastic Compute
Load Balancing
Autoscaling

AWS - SQS
Simple Queuing
System

SQS
- First in first out (FIFO)
- Single consumer per message
- Message has “visibility timeout”
- Adjustable visibility timeout

AWS - S3
File Storage

S3 - File Storage
- Store originals + output files
- Purge content after 7 days

AWS - DynamoDB

AWS - DynamoDB
AWS - MySQL on RDS

DynamoDB
- NoSQL
- Key-value pair storage
- Managed hosting
- Pay per record retrieval throughput

DynamoDB
- Our use case was not optimal
- No aggregations = no stats
- Higher cost than expected
- Doesn’t scale automatically

MySQL
- Switched to MySQL on RDS
- Managed hosting
- Aggregations!
- Data structure ported as-is

System Overview:
Autoscaling
Transcoders

How?
When?

Scaling Considerations
- Hourly billing
- Instance size: Cost vs Speed
- Content priority
- Ad-hoc requests
- Varied content length

AWS limitations
- AWS scaling metrics can be slow
- No automatic off switch
- Can’t choose instances to scale down
- (Instance protection now available)

so many variables :(

MVP:
Driven by profiles

Amazon Machine
Images (AMIs)

Which instance size?

Time vs Cost
for 1 hour of content
Time Cost
?

Winner: c4.xlarge

Up: step scaling
Down: stop everything

Interesting puzzle :)

System Overview:
Costs

24 hours in
Transcoding

(10am - 10am)

$57.22

$42.86
US Dollars

EC2 Instances: $34.31
EC2 (other): $ 4.24
RDS: $ 2.66
S3: $ 1.65
SQS: $ 0.00
Total: $42.86

Next 24 hours
Approx 70% of the volume

$25.94
US Dollars

One year’s costs?

$16.5k
Australian Dollars

($12.5k)
US Dollars

bikesandbeyond.ca

Future Roadmap:
Cost
Optimisations

Priority Content

On demand: 29c / hour
Spot: 4c / hour
c4.xlarge

Transcoding in
chunks/segments

Future Roadmap:
Introspection &
Validation

ffprobe

Future Roadmap:
Features

Adoption

Summary

We had opportunity
for a new system

www.iridetheharlemline.com/2013/05/13/a-commuters-rainbow

http://developers.digital.abc.net.au/
we’re hiring :-)

@daphnechong
@nickmchardy

Inside the ABC's new Media Transcoding system, Metro

More Related Content

What's hot

Viewers also liked

Similar to Inside the ABC's new Media Transcoding system, Metro

Recently uploaded

Inside the ABC's new Media Transcoding system, Metro