Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Video Transcoding at the ABC with Microservices at GOTO Chicago

260 views

Published on

ABC iview is a video on demand service from the Australian Broadcasting Corporation. In 2015, we built a new service in-house to handle transcoding for all of iview's content.

Metro is built on AWS, and uses FFmpeg, node.js and go components to transcode content quickly and cost-efficiently.

Presented at GOTO Chicago on 2nd May 2017

Published in: Technology
  • Be the first to comment

Video Transcoding at the ABC with Microservices at GOTO Chicago

  1. 1. Video Transcoding at the ABC with microservices Daphne Chong @daphnechong
  2. 2. media transcoder
  3. 3. 1. What is Transcoding? 2. Architecture 3. Autoscaling transcoders 4. Costs 5. Future roadmap
  4. 4. 1. What is Transcoding? 2. Architecture 3. Autoscaling transcoders 4. Costs 5. Future roadmap
  5. 5. What changes are we making?
  6. 6. High speed Medium speed Low speed
  7. 7. Standard Encoding H.264 MP4
  8. 8. Transcoding Example
  9. 9. Step 1: Prepare watermark
  10. 10. Step 1: Prepare watermark
  11. 11. Step 1: Prepare watermark
  12. 12. Step 2: Input video (GXF)
  13. 13. Step 3: Asymmetrical cropping
  14. 14. Step 4: Scale and set aspect ratio
  15. 15. Step 5: Apply watermark
  16. 16. Step 6: Audio Normalization
  17. 17. Original file audio track
  18. 18. After audio normalization
  19. 19. Step 7: Combine changes
  20. 20. Step 8: Split into segments
  21. 21. That’s a lot of changes...
  22. 22. Transcoding Profiles
  23. 23. GXF Profile: - Renditions (6) - Crop - Normalize audio - Adjust ratio - Add watermark - GXF to MP4
  24. 24. GXF
  25. 25. GXF MOV Profile: - Renditions (4) - Adjust ratio - ProRes MOV to MP4
  26. 26. GXF MOV
  27. 27. ./ffmpeg -i input.gxf output.mp4
  28. 28. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  29. 29. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  30. 30. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  31. 31. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  32. 32. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  33. 33. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  34. 34. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  35. 35. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  36. 36. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  37. 37. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  38. 38. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  39. 39. 1. What is Transcoding? 2. Architecture 3. Autoscaling transcoders 4. Costs 5. Future roadmap
  40. 40. 1. What is Transcoding? 2. Architecture 3. Autoscaling transcoders 4. Costs 5. Future roadmap
  41. 41. Workflow
  42. 42. 8 minutes 27 seconds
  43. 43. Original length Time to produce rendition 60 min 7 min 42 min Lowest Quality Highest Quality
  44. 44. AWS & Languages
  45. 45. Languages: Node.js Go
  46. 46. RDS - state storage EC2 - instances SQS - messaging S3 - file storage SNS - notifications AWS Infrastructure:
  47. 47. Advantages Deployments Independent scaling Really easy to expand Clear responsibilities
  48. 48. Challenges Deployments General overhead
  49. 49. 1. What is Transcoding? 2. Architecture 3. Autoscaling transcoders 4. Costs 5. Future optimizations
  50. 50. 1. What is Transcoding? 2. Architecture 3. Autoscaling transcoders 4. Costs 5. Future optimizations
  51. 51. When should you scale?
  52. 52. Scaling Considerations: Content priority Ad-hoc requests Varied content length
  53. 53. AWS Considerations: Hourly billing Cost vs Speed Fleet reaction time Scaling down
  54. 54. MVP: Don’t make things wait too long
  55. 55. MVP: Supply enough capacity at all times
  56. 56. GXF Profile: - Renditions (6) - Crop - Normalize audio - Adjust ratio - Add watermark - GXF to MP4
  57. 57. var autoscaling = new AWS.AutoScaling(...); var params = { AutoScalingGroupName: groupName, DesiredCapacity: capacity Some white text }; autoscaling.setDesiredCapacity(params, ...);
  58. 58. var autoscaling = new AWS.AutoScaling(...); var params = { AutoScalingGroupName: groupName, DesiredCapacity: capacity Some white text }; autoscaling.setDesiredCapacity(params, ...);
  59. 59. Which instance size?
  60. 60. Time vs Cost for 1 hour of content Time Cost ?
  61. 61. Time vs Cost for 1 hour of content Time Cost ?
  62. 62. Winner: c4.xlarge
  63. 63. When to scale down?
  64. 64. Up: step scaling Down: stop everything
  65. 65. An interesting puzzle :)
  66. 66. 1. What is Transcoding? 2. Architecture 3. Autoscaling transcoders 4. Costs 5. Future optimizations
  67. 67. 1. What is Transcoding? 2. Architecture 3. Autoscaling transcoders 4. Costs 5. Future optimizations
  68. 68. What about commercial transcode providers?
  69. 69. Full-Service Transcoding
  70. 70. $(a lot)
  71. 71. DIY on-demand
  72. 72. $45k
  73. 73. Metro’s costs?
  74. 74. $11.2k
  75. 75. Some daily examples
  76. 76. (10am - 10am)
  77. 77. $42.86
  78. 78. EC2 Instances: $34.31 EC2 (other): $ 4.24 RDS: $ 2.66 S3: $ 1.65 SQS: $ 0.00 Total (USD): $42.86 Transcoding costs: 24 hrs
  79. 79. Next 24 hours
  80. 80. $25.93
  81. 81. 1. What is Transcoding? 2. Architecture 3. Autoscaling transcoders 4. Costs 5. Future optimizations
  82. 82. 1. What is Transcoding? 2. Architecture 3. Autoscaling transcoders 4. Costs 5. Future optimizations
  83. 83. Throughput optimizations
  84. 84. Transcoding in chunks/segments
  85. 85. Introspection & Validation
  86. 86. GXF Profile: - Crop - Normalize audio - Adjust ratio - Add watermark - GXF to MP4
  87. 87. ffprobe
  88. 88. 1. What is Transcoding? 2. Architecture 3. Autoscaling transcoders 4. Costs 5. Future optimizations
  89. 89. Summary
  90. 90. 8 minutes 27 seconds
  91. 91. $11.2k
  92. 92. @daphnechong Thanks!

×