Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Inside the ABC's new Media Transcoding system, Metro

6,238 views

Published on

The ABC recently launched a new video transcoding system to process all the video content for ABC iview, our catch-up TV service.

Metro is a cost-efficient, scalable, cloud-based system that was built using Golang, Node, FFmpeg, and heavily utilises a variety of AWS technology including queues, varied capacity autoscaling, hosted database servers, and notifications. The system has been live since December 2015, and has successfully processed thousands of pieces of content.

Published in: Technology
  • Be the first to comment

Inside the ABC's new Media Transcoding system, Metro

  1. 1. metro - Media Transcoder metro
  2. 2. metro - Media Transcoder Daphne Chong Nick McHardy @daphnechong @nickmchardy
  3. 3. metro - Media Transcoder Questions?
  4. 4. metro - Media Transcoder Project Background - What is Transcoding? - Examples - FFmpeg - Why did we build our own?
  5. 5. metro - Media Transcoder Architecture - Architecture overview - Technology choices - Transcoder autoscaling policies - Costs
  6. 6. metro - Media Transcoder Future Roadmap - Cost optimisations - Validation and Introspection - Features
  7. 7. metro - Media Transcoder Project Background: What is Transcoding?
  8. 8. metro - Media Transcoder
  9. 9. metro - Media Transcoder Converting video to different formats
  10. 10. metro - Media Transcoder
  11. 11. metro - Media Transcoder
  12. 12. metro - Media Transcoder High speed Medium speed Low speed
  13. 13. metro - Media Transcoder Standard Encoding
  14. 14. metro - Media Transcoder Project Background: Transcoding Example
  15. 15. metro - Media Transcoder Step 1: Prepare watermark
  16. 16. metro - Media Transcoder Step 1: Prepare watermark
  17. 17. metro - Media Transcoder Step 1: Prepare watermark
  18. 18. metro - Media Transcoder Step 2: Input video (GXF)
  19. 19. metro - Media Transcoder Step 3: Asymmetrical cropping
  20. 20. metro - Media Transcoder Step 4: Scale and set aspect ratio
  21. 21. metro - Media Transcoder Step 5: Apply watermark
  22. 22. metro - Media Transcoder Step 6: Ready to stream 1500k
  23. 23. metro - Media Transcoder other rendition sizes 1000k 650k 500k 220k audio only
  24. 24. metro - Media Transcoder Original file audio track
  25. 25. metro - Media Transcoder Previous transcoder audio track
  26. 26. metro - Media Transcoder After audio normalisation
  27. 27. metro - Media Transcoder Transcoding profiles
  28. 28. metro - Media Transcoder GXF, ABC1 Profile: - Crop - Normalise audio - Adjust ratio - Add watermark - GXF to MP4
  29. 29. metro - Media Transcoder GXF, ABC1
  30. 30. metro - Media Transcoder GXF, ABC1 MOV, Arts Profile: - Adjust ratio - ProRes MOV to MP4
  31. 31. metro - Media Transcoder GXF, ABC1 MOV, Arts
  32. 32. metro - Media Transcoder Project Background: FFmpeg
  33. 33. metro - Media Transcoder ./ffmpeg utopia.gxf utopia.mp4
  34. 34. metro - Media Transcoder "hi+": { "transcoder": "ffmpeg", "transcoderoptions": "-c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif, scale=800x450,setdar=16/9[main], [1:v]scale=42:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-37: main_h-overlay_h-34" -crf 22 -maxrate 904k -bufsize 1000k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y", "bitrate": "1000k", "outputfilemimetype": "video/mp4" }, "hi++": { "transcoder": "ffmpeg", "transcoderoptions": "-c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif, scale=1024x576,setdar=16/9[main], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47: main_h-overlay_h-44" -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high - level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y", "bitrate": "1500k", "outputfilemimetype": "video/mp4" } FFmpeg filters are fun
  35. 35. metro - Media Transcoder af_sidhaiompress f_senmd vfropdett vf_kerndeint vf_signalstats af_adelay af_sileedett f_zmq vfurves vf_lenorrtion vf_smartblur af_aho af_sileeremove fifo vf_tdnoiz vf_libopev vf_spp af_aemphasis af_sofalizer filtfmts vf_deband vf_lut vf_ssim af_afade af_stereotools formats vf_dimate vf_lut3d vf_stk af_aformat af_stereowiden framepool vf_dejudder vf_maskedmerge vf_stereo3d af_agate af_tremolo framesy vf_delogo vf_deint vf_subtitles af_alimiter af_vibrato generate_wave_table vf_deshake vf_mergeplanes vf_super2xsai af_amerge af_volume graphdump vf_deteline vf_mpdimate vf_swapuv af_amix af_volumedett graphparser vf_disple vf_neighbor vf_teline af_anequalizer allfilters lavfutils vf_drawbox vf_noise vf_thumbnail af_anull asink_anullsink log2_tab vf_drawtext vf_null vf_tile af_apad as_anoises lswsutils vf_edgedett vf_r vf_tinterle af_aphaser as_anulls opel_allkernels vf_elbg vf_overlay vf_transpose af_apulsator as_flite pthread vf_eq vf_owdenoise vf_unsharp af_aresample as_sine setpts vf_extrtplanes vf_pad vf_uspp af_asetnsamples audio settb vf_fade vf_palettegen vf_vtorope af_asetrate avf_aphasemeter split vf_fftfilt vf_paletteuse vf_vflip af_ashowinfo avf_avtorope s_movie vf_field vf_persptive vf_vidstabdett af_astats avfoat transform vf_fieldmah vf_phase vf_vidstabtransform af_asyts avf_shoqt trim vf_fieldorder vf_pixdetest vf_vignette af_atempo avf_showfreqs unsharp_opel vf_find_rt vf_pp vf_w3fdif af_biquads avf_showsptrum vf_alphamerge vf_format vf_pp7 vf_waveform af_bs2b avf_showvolume vf_aspt vf_fps vf_psnr vf_xbr afhannelmap avf_showwaves vf_atadenoise vf_framepk vf_pullup vf_yadif afhannelsplit avfilter vf_bbox vf_framerate vf_qp vf_zoompan afhorus avfiltergraph vf_blkdett vf_framestep vf_random vf_zale afompand bbox vf_blkframe vf_frei0r vf_removegrain video afompensationdelay buffersink vf_blend vf_fspp vf_removelogo vidstabutils af_shift buffers vf_boxblur vf_geq vf_repeatfields vsink_nullsink af_dynaudnorm deshake_opel vfhromakey vf_gradfun vf_rotate vsellauto af_earwax drawutils vfodview vf_hflip vf_sab vs_life af_extrastereo dualinput vfolorbalae vf_histeq vf_ale vs_mandelbrot af_flanger f_drawgraph vfolohannelmixer vf_histogram vf_seltivolor vs_mptests af_join f_ebur128 vfolorkey vf_hqdn3d vf_separatefields vs_tests af_ladspa f_interleave vfolorlevels vf_hqx vf_setfield window_fu af_pan f_perms vfolormatrix vf_hue vf_showinfo af_replaygain f_realtime vfopy vf_idet vf_showpalette af_resample f_reverse vfover_rt vf_il vf_shuffleframes af_rubberband f_selt vfrop vf_interle vf_shuffleplanes other FFmpeg filters
  36. 36. metro - Media Transcoder
  37. 37. metro - Media Transcoder
  38. 38. metro - Media Transcoder High speed Medium speed Low speed
  39. 39. metro - Media Transcoder Step 1: Prepare watermark
  40. 40. metro - Media Transcoder After audio normalisation
  41. 41. metro - Media Transcoder GXF, ABC1 Profile: - Crop - Normalise audio - Adjust ratio - Add watermark - GXF to MP4
  42. 42. metro - Media Transcoder ./ffmpeg utopia.gxf utopia.mp4
  43. 43. metro - Media Transcoder Project Background: Why did we build Metro?
  44. 44. metro - Media Transcoder
  45. 45. metro - Media Transcoder $$$
  46. 46. metro - Media Transcoder thesupercarkids.com
  47. 47. metro - Media Transcoder mini-me.com
  48. 48. metro - Media Transcoder Thousands of pieces of content
  49. 49. metro - Media Transcoder More customisation & flexibility
  50. 50. metro - Media Transcoder
  51. 51. metro - Media Transcoder Cost vs Speed Scale Flexibility
  52. 52. metro - Media Transcoder Questions?
  53. 53. metro - Media Transcoder System Overview: Architecture
  54. 54. metro - Media Transcoder “The Cloud”
  55. 55. metro - Media Transcoder
  56. 56. metro - Media Transcoder
  57. 57. metro - Media Transcoder
  58. 58. metro - Media Transcoder
  59. 59. metro - Media Transcoder
  60. 60. metro - Media Transcoder
  61. 61. metro - Media Transcoder
  62. 62. metro - Media Transcoder
  63. 63. metro - Media Transcoder
  64. 64. metro - Media Transcoder
  65. 65. metro - Media Transcoder
  66. 66. metro - Media Transcoder
  67. 67. metro - Media Transcoder
  68. 68. metro - Media Transcoder
  69. 69. metro - Media Transcoder
  70. 70. metro - Media Transcoder
  71. 71. metro - Media Transcoder 8 minutes 27 seconds
  72. 72. metro - Media Transcoder Original length Time to produce rendition 60 min 7 min 42 min Lowest (audio) Highest (1500k)
  73. 73. metro - Media Transcoder
  74. 74. metro - Media Transcoder System Overview: Technology Choices
  75. 75. metro - Media Transcoder Node + Go - AWS: EC2, SQS, - S3, Dynamo/RDS
  76. 76. metro - Media Transcoder Node.js Components
  77. 77. metro - Media Transcoder
  78. 78. metro - Media Transcoder Node.js - Existing team experience - Rapid development - Solid choice for APIs - Reservations for long lived processes - Untyped. Sadface.
  79. 79. metro - Media Transcoder Golang Components
  80. 80. metro - Media Transcoder
  81. 81. metro - Media Transcoder Golang - Typed and compiled - Cross-platform - Easy deployment - Concurrency (we don’t use this heavily)
  82. 82. metro - Media Transcoder Golang - Reasonable learning curve - Magic interfaces - Different techniques for testing - Dependencies are challenging
  83. 83. metro - Media Transcoder AWS - EC2 Elastic Compute Load Balancing Autoscaling
  84. 84. metro - Media Transcoder
  85. 85. metro - Media Transcoder AWS - SQS Simple Queuing System
  86. 86. metro - Media Transcoder
  87. 87. metro - Media Transcoder SQS - First in first out (FIFO) - Single consumer per message - Message has “visibility timeout” - Adjustable visibility timeout
  88. 88. metro - Media Transcoder AWS - S3 File Storage
  89. 89. metro - Media Transcoder
  90. 90. metro - Media Transcoder S3 - File Storage - Store originals + output files - Purge content after 7 days
  91. 91. metro - Media Transcoder AWS - DynamoDB
  92. 92. metro - Media Transcoder AWS - DynamoDB
  93. 93. metro - Media Transcoder AWS - DynamoDB AWS - MySQL on RDS
  94. 94. metro - Media Transcoder
  95. 95. metro - Media Transcoder DynamoDB - NoSQL - Key-value pair storage - Managed hosting - Pay per record retrieval throughput
  96. 96. metro - Media Transcoder DynamoDB - Our use case was not optimal - No aggregations = no stats - Higher cost than expected - Doesn’t scale automatically
  97. 97. metro - Media Transcoder MySQL - Switched to MySQL on RDS - Managed hosting - Aggregations! - Data structure ported as-is
  98. 98. metro - Media Transcoder Node + Go - AWS: EC2, SQS, - S3, Dynamo/RDS
  99. 99. metro - Media Transcoder System Overview: Autoscaling Transcoders
  100. 100. metro - Media Transcoder
  101. 101. metro - Media Transcoder How? When?
  102. 102. metro - Media Transcoder Scaling Considerations - Hourly billing - Instance size: Cost vs Speed - Content priority - Ad-hoc requests - Varied content length
  103. 103. metro - Media Transcoder AWS limitations - AWS scaling metrics can be slow - No automatic off switch - Can’t choose instances to scale down - (Instance protection now available)
  104. 104. metro - Media Transcoder so many variables :(
  105. 105. metro - Media Transcoder MVP: Driven by profiles
  106. 106. metro - Media Transcoder Amazon Machine Images (AMIs)
  107. 107. metro - Media Transcoder Which instance size?
  108. 108. metro - Media Transcoder Time vs Cost for 1 hour of content Time Cost ?
  109. 109. metro - Media Transcoder Winner: c4.xlarge
  110. 110. metro - Media Transcoder Up: step scaling Down: stop everything
  111. 111. metro - Media Transcoder
  112. 112. metro - Media Transcoder
  113. 113. metro - Media Transcoder
  114. 114. metro - Media Transcoder
  115. 115. metro - Media Transcoder
  116. 116. metro - Media Transcoder Interesting puzzle :)
  117. 117. metro - Media Transcoder Questions?
  118. 118. metro - Media Transcoder System Overview: Costs
  119. 119. metro - Media Transcoder 24 hours in Transcoding
  120. 120. metro - Media Transcoder (10am - 10am)
  121. 121. metro - Media Transcoder $57.22
  122. 122. metro - Media Transcoder $42.86 US Dollars
  123. 123. metro - Media Transcoder EC2 Instances: $34.31 EC2 (other): $ 4.24 RDS: $ 2.66 S3: $ 1.65 SQS: $ 0.00 Total: $42.86
  124. 124. metro - Media Transcoder Next 24 hours Approx 70% of the volume
  125. 125. metro - Media Transcoder $25.94 US Dollars
  126. 126. metro - Media Transcoder One year’s costs?
  127. 127. metro - Media Transcoder $16.5k Australian Dollars
  128. 128. metro - Media Transcoder ($12.5k) US Dollars
  129. 129. metro - Media Transcoder thesupercarkids.com
  130. 130. metro - Media Transcoder mini-me.com
  131. 131. metro - Media Transcoder bikesandbeyond.ca
  132. 132. metro - Media Transcoder Future Roadmap: Cost Optimisations
  133. 133. metro - Media Transcoder Priority Content
  134. 134. metro - Media Transcoder
  135. 135. metro - Media Transcoder
  136. 136. metro - Media Transcoder On demand: 29c / hour Spot: 4c / hour c4.xlarge
  137. 137. metro - Media Transcoder Transcoding in chunks/segments
  138. 138. metro - Media Transcoder
  139. 139. metro - Media Transcoder Future Roadmap: Introspection & Validation
  140. 140. metro - Media Transcoder GXF, ABC1 Profile: - Crop - Normalise audio - Adjust ratio - Add watermark - GXF to MP4
  141. 141. metro - Media Transcoder ffprobe
  142. 142. metro - Media Transcoder
  143. 143. metro - Media Transcoder
  144. 144. metro - Media Transcoder
  145. 145. metro - Media Transcoder
  146. 146. metro - Media Transcoder
  147. 147. metro - Media Transcoder Future Roadmap: Features
  148. 148. metro - Media Transcoder
  149. 149. metro - Media Transcoder
  150. 150. metro - Media Transcoder
  151. 151. metro - Media Transcoder
  152. 152. metro - Media Transcoder Adoption
  153. 153. metro - Media Transcoder Summary
  154. 154. metro - Media Transcoder We had opportunity for a new system
  155. 155. metro - Media Transcoder
  156. 156. metro - Media Transcoder
  157. 157. metro - Media Transcoder
  158. 158. metro - Media Transcoder
  159. 159. metro - Media Transcoder 8 minutes 27 seconds
  160. 160. metro - Media Transcoder bikesandbeyond.ca
  161. 161. metro - Media Transcoder www.iridetheharlemline.com/2013/05/13/a-commuters-rainbow
  162. 162. metro - Media Transcoder http://developers.digital.abc.net.au/ we’re hiring :-)
  163. 163. metro - Media Transcoder @daphnechong @nickmchardy
  164. 164. metro - Media Transcoder Q&A

×