Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
metro - Media Transcoder
metro
metro - Media Transcoder
metro
metro - Media Transcoder
Daphne Chong
Nick McHardy
@daphnechong
@nickmchardy
metro - Media Transcoder
Things we’ll cover:
1. What is Transcoding?
2. Why build our own system?
3. Architecture & Tech
4...
metro - Media Transcoder
Things we’ll cover:
1. What is Transcoding?
2. Why build our own system?
3. Architecture & Tech
4...
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
High speed
Medium speed
Low speed
metro - Media Transcoder
Standard Encoding:
H.264 mpeg4
metro - Media Transcoder
Step 1: Input video (GXF)
metro - Media Transcoder
Step 2: Asymmetrical cropping
metro - Media Transcoder
Step 3: Scale and set aspect ratio
metro - Media Transcoder
Step 4: Apply watermark
metro - Media Transcoder
Step 5: Ready to stream
1500k
metro - Media Transcoder
other rendition sizes
1000k
650k
500k
220k
audio only
metro - Media Transcoder
GXF, ABC1
Profile:
- Renditions (2)
- Crop
- Normalise audio
- Adjust ratio
- Add watermark
- GXF...
metro - Media Transcoder
GXF, ABC1
metro - Media Transcoder
GXF, ABC1
MOV, Arts
Profile:
- Adjust ratio
- ProRes MOV to MP4
metro - Media Transcoder
GXF, ABC1
MOV, Arts
metro - Media Transcoder
1. What is Transcoding?
2. Why build our own system?
3. Architecture & Tech
4. Autoscaling transc...
metro - Media Transcoder
1. What is Transcoding?
2. Why build our own system?
3. Architecture & Tech
4. Autoscaling transc...
metro - Media Transcoder
Cost vs Speed
Scale
Flexibility
metro - Media Transcoder
Cost vs Speed
Scale
Flexibility
metro - Media Transcoder
metro - Media Transcoder
$$$
metro - Media Transcoder
thesupercarkids.com
metro - Media Transcoder
mini-me.com
metro - Media Transcoder
Cost vs Speed
Scale
Flexibility
metro - Media Transcoder
Thousands of
pieces of content
metro - Media Transcoder
Cost vs Speed
Scale
Flexibility
metro - Media Transcoder
metro - Media Transcoder
Cost vs Speed
Scale
Flexibility
metro - Media Transcoder
1. What is Transcoding?
2. Why build our own system?
3. Architecture & Tech
4. Autoscaling transc...
metro - Media Transcoder
1. What is Transcoding?
2. Why build our own system?
3. Architecture & Tech
4. Autoscaling transc...
metro - Media Transcoder
“The Cloud”
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
8 minutes 27 seconds
metro - Media Transcoder
Languages
and Software
metro - Media Transcoder
Node.js
Components
metro - Media Transcoder
metro - Media Transcoder
Node.js
- Existing team experience
- Rapid development for APIs
- Untyped. Sadface.
metro - Media Transcoder
Golang
Components
metro - Media Transcoder
metro - Media Transcoder
Golang
- Typed and compiled
- Cross-platform
- Easy deployment
- Learning curve
metro - Media Transcoder
FFmpeg
metro - Media Transcoder
./ffmpeg utopia.gxf utopia.mp4
metro - Media Transcoder
"hi+": {
"transcoder": "ffmpeg",
"transcoderoptions": "-c:v libx264 -filter_complex "[0:v]crop=72...
metro - Media Transcoder
AWS
EC2 Autoscaling
metro - Media Transcoder
metro - Media Transcoder
AWS:
Simple Queuing
System (SQS)
metro - Media Transcoder
metro - Media Transcoder
1. What is Transcoding?
2. Why build our own system?
3. Architecture & Tech
4. Autoscaling transc...
metro - Media Transcoder
1. What is Transcoding?
2. Why build our own system?
3. Architecture & Tech
4. Autoscaling transc...
metro - Media Transcoder
metro - Media Transcoder
How?
When?
metro - Media Transcoder
Scaling Considerations
- Hourly billing
- Instance size: Cost vs Speed
- Content priority
- Ad-ho...
metro - Media Transcoder
So many variables :(
metro - Media Transcoder
MVP:
Driven by drop folders
metro - Media Transcoder
GXF, ABC1
Profile:
- Renditions (2)
- Crop
- Normalise audio
- Adjust ratio
- Add watermark
- GXF...
metro - Media Transcoder
Which instance size?
metro - Media Transcoder
Time vs Cost
for 1 hour of content
Time Cost
?
metro - Media Transcoder
Winner: c4.xlarge
metro - Media Transcoder
Up: step scaling
Down: stop everything
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
Interesting puzzle :)
metro - Media Transcoder
1. What is Transcoding?
2. Why build our own system?
3. Architecture & Tech
4. Autoscaling transc...
metro - Media Transcoder
1. What is Transcoding?
2. Why build our own system?
3. Architecture & Tech
4. Autoscaling transc...
metro - Media Transcoder
24 hours in
Transcoding
metro - Media Transcoder
(10am - 10am)
metro - Media Transcoder
$57.22
metro - Media Transcoder
$42.86
US Dollars
metro - Media Transcoder
EC2 Instances: $34.31
EC2 (other): $ 4.24
RDS: $ 2.66
S3: $ 1.65
SQS: $ 0.00
Total (USD): $42.86
metro - Media Transcoder
Next 24 hours
Approx 70% of the volume
metro - Media Transcoder
$25.94
US Dollars
metro - Media Transcoder
One year’s costs?
metro - Media Transcoder
$16.5k
Australian Dollars
metro - Media Transcoder
$12.5k
US Dollars
metro - Media Transcoder
thesupercarkids.com
metro - Media Transcoder
mini-me.com
metro - Media Transcoder
bikesandbeyond.ca
metro - Media Transcoder
Summary
metro - Media Transcoder
We had opportunity
for a new system
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
metro - Media Transcoder
8 minutes 27 seconds
metro - Media Transcoder
$$$
metro - Media Transcoder
http://developers.digital.abc.net.au/
we’re hiring :-)
metro - Media Transcoder
@daphnechong
@nickmchardy
metro - Media Transcoder
Q&A
Upcoming SlideShare
Loading in …5
×

YOW! West 2016: Inside the ABC's new Media Transcoding service, Metro

537 views

Published on

The ABC recently launched a new video transcoding system to process all the video content for ABC iview, our catch-up TV service.

Metro is a cost-efficient, scalable, cloud-based system that was built using Golang, Node, FFmpeg, and heavily utilises a variety of AWS technology including queues, varied capacity autoscaling, hosted database servers, and notifications. The system has been live since December 2015, and has successfully processed thousands of pieces of content.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

YOW! West 2016: Inside the ABC's new Media Transcoding service, Metro

  1. 1. metro - Media Transcoder metro
  2. 2. metro - Media Transcoder metro
  3. 3. metro - Media Transcoder Daphne Chong Nick McHardy @daphnechong @nickmchardy
  4. 4. metro - Media Transcoder Things we’ll cover: 1. What is Transcoding? 2. Why build our own system? 3. Architecture & Tech 4. Autoscaling transcoders 5. Costs
  5. 5. metro - Media Transcoder Things we’ll cover: 1. What is Transcoding? 2. Why build our own system? 3. Architecture & Tech 4. Autoscaling transcoders 5. Costs
  6. 6. metro - Media Transcoder
  7. 7. metro - Media Transcoder
  8. 8. metro - Media Transcoder
  9. 9. metro - Media Transcoder High speed Medium speed Low speed
  10. 10. metro - Media Transcoder Standard Encoding: H.264 mpeg4
  11. 11. metro - Media Transcoder Step 1: Input video (GXF)
  12. 12. metro - Media Transcoder Step 2: Asymmetrical cropping
  13. 13. metro - Media Transcoder Step 3: Scale and set aspect ratio
  14. 14. metro - Media Transcoder Step 4: Apply watermark
  15. 15. metro - Media Transcoder Step 5: Ready to stream 1500k
  16. 16. metro - Media Transcoder other rendition sizes 1000k 650k 500k 220k audio only
  17. 17. metro - Media Transcoder GXF, ABC1 Profile: - Renditions (2) - Crop - Normalise audio - Adjust ratio - Add watermark - GXF to MP4
  18. 18. metro - Media Transcoder GXF, ABC1
  19. 19. metro - Media Transcoder GXF, ABC1 MOV, Arts Profile: - Adjust ratio - ProRes MOV to MP4
  20. 20. metro - Media Transcoder GXF, ABC1 MOV, Arts
  21. 21. metro - Media Transcoder 1. What is Transcoding? 2. Why build our own system? 3. Architecture & Tech 4. Autoscaling transcoders 5. Costs
  22. 22. metro - Media Transcoder 1. What is Transcoding? 2. Why build our own system? 3. Architecture & Tech 4. Autoscaling transcoders 5. Costs
  23. 23. metro - Media Transcoder Cost vs Speed Scale Flexibility
  24. 24. metro - Media Transcoder Cost vs Speed Scale Flexibility
  25. 25. metro - Media Transcoder
  26. 26. metro - Media Transcoder $$$
  27. 27. metro - Media Transcoder thesupercarkids.com
  28. 28. metro - Media Transcoder mini-me.com
  29. 29. metro - Media Transcoder Cost vs Speed Scale Flexibility
  30. 30. metro - Media Transcoder Thousands of pieces of content
  31. 31. metro - Media Transcoder Cost vs Speed Scale Flexibility
  32. 32. metro - Media Transcoder
  33. 33. metro - Media Transcoder Cost vs Speed Scale Flexibility
  34. 34. metro - Media Transcoder 1. What is Transcoding? 2. Why build our own system? 3. Architecture & Tech 4. Autoscaling transcoders 5. Costs
  35. 35. metro - Media Transcoder 1. What is Transcoding? 2. Why build our own system? 3. Architecture & Tech 4. Autoscaling transcoders 5. Costs
  36. 36. metro - Media Transcoder “The Cloud”
  37. 37. metro - Media Transcoder
  38. 38. metro - Media Transcoder
  39. 39. metro - Media Transcoder
  40. 40. metro - Media Transcoder
  41. 41. metro - Media Transcoder
  42. 42. metro - Media Transcoder
  43. 43. metro - Media Transcoder
  44. 44. metro - Media Transcoder
  45. 45. metro - Media Transcoder
  46. 46. metro - Media Transcoder
  47. 47. metro - Media Transcoder
  48. 48. metro - Media Transcoder
  49. 49. metro - Media Transcoder
  50. 50. metro - Media Transcoder
  51. 51. metro - Media Transcoder
  52. 52. metro - Media Transcoder
  53. 53. metro - Media Transcoder 8 minutes 27 seconds
  54. 54. metro - Media Transcoder Languages and Software
  55. 55. metro - Media Transcoder Node.js Components
  56. 56. metro - Media Transcoder
  57. 57. metro - Media Transcoder Node.js - Existing team experience - Rapid development for APIs - Untyped. Sadface.
  58. 58. metro - Media Transcoder Golang Components
  59. 59. metro - Media Transcoder
  60. 60. metro - Media Transcoder Golang - Typed and compiled - Cross-platform - Easy deployment - Learning curve
  61. 61. metro - Media Transcoder FFmpeg
  62. 62. metro - Media Transcoder ./ffmpeg utopia.gxf utopia.mp4
  63. 63. metro - Media Transcoder "hi+": { "transcoder": "ffmpeg", "transcoderoptions": "-c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif, scale=800x450,setdar=16/9[main], [1:v]scale=42:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-37: main_h-overlay_h-34" -crf 22 -maxrate 904k -bufsize 1000k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y", "bitrate": "1000k", "outputfilemimetype": "video/mp4" }, "hi++": { "transcoder": "ffmpeg", "transcoderoptions": "-c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif, scale=1024x576,setdar=16/9[main], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47: main_h-overlay_h-44" -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high - level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y", "bitrate": "1500k", "outputfilemimetype": "video/mp4" } FFmpeg filters are fun
  64. 64. metro - Media Transcoder AWS EC2 Autoscaling
  65. 65. metro - Media Transcoder
  66. 66. metro - Media Transcoder AWS: Simple Queuing System (SQS)
  67. 67. metro - Media Transcoder
  68. 68. metro - Media Transcoder 1. What is Transcoding? 2. Why build our own system? 3. Architecture & Tech 4. Autoscaling transcoders 5. Costs
  69. 69. metro - Media Transcoder 1. What is Transcoding? 2. Why build our own system? 3. Architecture & Tech 4. Autoscaling transcoders 5. Costs
  70. 70. metro - Media Transcoder
  71. 71. metro - Media Transcoder How? When?
  72. 72. metro - Media Transcoder Scaling Considerations - Hourly billing - Instance size: Cost vs Speed - Content priority - Ad-hoc requests - Varied content length
  73. 73. metro - Media Transcoder So many variables :(
  74. 74. metro - Media Transcoder MVP: Driven by drop folders
  75. 75. metro - Media Transcoder GXF, ABC1 Profile: - Renditions (2) - Crop - Normalise audio - Adjust ratio - Add watermark - GXF to MP4
  76. 76. metro - Media Transcoder Which instance size?
  77. 77. metro - Media Transcoder Time vs Cost for 1 hour of content Time Cost ?
  78. 78. metro - Media Transcoder Winner: c4.xlarge
  79. 79. metro - Media Transcoder Up: step scaling Down: stop everything
  80. 80. metro - Media Transcoder
  81. 81. metro - Media Transcoder
  82. 82. metro - Media Transcoder
  83. 83. metro - Media Transcoder
  84. 84. metro - Media Transcoder
  85. 85. metro - Media Transcoder Interesting puzzle :)
  86. 86. metro - Media Transcoder 1. What is Transcoding? 2. Why build our own system? 3. Architecture & Tech 4. Autoscaling transcoders 5. Costs
  87. 87. metro - Media Transcoder 1. What is Transcoding? 2. Why build our own system? 3. Architecture & Tech 4. Autoscaling transcoders 5. Costs
  88. 88. metro - Media Transcoder 24 hours in Transcoding
  89. 89. metro - Media Transcoder (10am - 10am)
  90. 90. metro - Media Transcoder $57.22
  91. 91. metro - Media Transcoder $42.86 US Dollars
  92. 92. metro - Media Transcoder EC2 Instances: $34.31 EC2 (other): $ 4.24 RDS: $ 2.66 S3: $ 1.65 SQS: $ 0.00 Total (USD): $42.86
  93. 93. metro - Media Transcoder Next 24 hours Approx 70% of the volume
  94. 94. metro - Media Transcoder $25.94 US Dollars
  95. 95. metro - Media Transcoder One year’s costs?
  96. 96. metro - Media Transcoder $16.5k Australian Dollars
  97. 97. metro - Media Transcoder $12.5k US Dollars
  98. 98. metro - Media Transcoder thesupercarkids.com
  99. 99. metro - Media Transcoder mini-me.com
  100. 100. metro - Media Transcoder bikesandbeyond.ca
  101. 101. metro - Media Transcoder Summary
  102. 102. metro - Media Transcoder We had opportunity for a new system
  103. 103. metro - Media Transcoder
  104. 104. metro - Media Transcoder
  105. 105. metro - Media Transcoder
  106. 106. metro - Media Transcoder 8 minutes 27 seconds
  107. 107. metro - Media Transcoder $$$
  108. 108. metro - Media Transcoder http://developers.digital.abc.net.au/ we’re hiring :-)
  109. 109. metro - Media Transcoder @daphnechong @nickmchardy
  110. 110. metro - Media Transcoder Q&A

×