Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Video Transcoding
at Scale for ABC iview
Daphne Chong
@daphnechong
metro
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
What changes
are we making?
High speed
Medium speed
Low speed
Standard
Encoding
H.264 MP4
Transcoding
Example
Step 1: Prepare watermark
Step 1: Prepare watermark
Step 1: Prepare watermark
Step 2: Input video (GXF)
Step 3: Asymmetrical cropping
Step 4: Scale and set aspect ratio
Step 5: Apply watermark
Step 6: Ready to stream
1500k
other rendition sizes
1000k
650k
500k
220k
audio only
Original file audio track
Previous transcoder audio track
After audio normalisation
That’s a lot of
changes...
Transcoding
Profiles
GXF, ABC1
Profile:
- Renditions (6)
- Crop
- Normalise audio
- Adjust ratio
- Add watermark
- GXF to MP4
GXF, ABC1
GXF, ABC1
MOV, Arts
Profile:
- Renditions (6)
- Adjust ratio
- ProRes MOV to MP4
GXF, ABC1
MOV, Arts
What is this
“Magic Video Software”
you speak of?
./ffmpeg -i utopia.gxf utopia.mp4
./ffmpeg -i INPUT -c:v libx264 -filter_complex
"[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma
in], [1:v]scale...
./ffmpeg -i INPUT -c:v libx264 -filter_complex
"[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma
in], [1:v]scale...
./ffmpeg -i INPUT -c:v libx264 -filter_complex
"[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma
in], [1:v]scale...
./ffmpeg -i INPUT -c:v libx264 -filter_complex
"[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma
in], [1:v]scale...
./ffmpeg -i INPUT -c:v libx264 -filter_complex
"[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma
in], [1:v]scale...
./ffmpeg -i INPUT -c:v libx264 -filter_complex
"[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma
in], [1:v]scale...
./ffmpeg -i INPUT -c:v libx264 -filter_complex
"[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma
in], [1:v]scale...
./ffmpeg -i INPUT -c:v libx264 -filter_complex
"[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma
in], [1:v]scale...
./ffmpeg -i INPUT -c:v libx264 -filter_complex
"[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma
in], [1:v]scale...
./ffmpeg -i INPUT -c:v libx264 -filter_complex
"[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma
in], [1:v]scale...
./ffmpeg -i INPUT -c:v libx264 -filter_complex
"[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma
in], [1:v]scale...
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
Cost
Scale
Flexibility
Cost
Scale
Flexibility
thesupercarkids.com
mini-me.com
Cost
Scale
Flexibility
“The Cloud”
Cost
Scale
Flexibility
Cost
Scale
Flexibility
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
Source
Queue
Process
Transfer
Notify
8 minutes 27 seconds
Original
length
Time to produce rendition
60 min
7 min
42 min
Lowest
(audio)
Highest
(1500k)
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
Languages:
Node.js
Go
Node.js
Existing team experience
Rapid development for APIs
Untyped. Sadface.
Go
Typed and compiled
Cross-platform
Easy deployment
Learning curve
Infrastructure:
EC2, SQS, S3, SNS,
Dynamo / RDS,
Cloudformation
AWS - EC2
Autoscaling
AWS - SQS
Simple Queuing
System
SQS
First in first out (FIFO)
Single consumer per message
Message has “visibility timeout”
Adjustable visibility timeout
AWS - S3
File Storage
AWS - DynamoDB
AWS - DynamoDB
AWS - DynamoDB
AWS - MySQL on RDS
DynamoDB
NoSQL
Key-value pair storage
Pay for number of records/sec read
DynamoDB
Our use case was not optimal
Manual aggregations
Higher cost than expected
Doesn’t scale automatically
MySQL
Switched to MySQL on RDS
Managed hosting
Aggregations!
Data structure ported as-is
AWS Cloudformation
"ExampleLoadBalancer" : {
"Type" : "AWS::ElasticLoadBalancing::LoadBalancer",
"Properties" : {
"AvailabilityZones" : [ "us...
Node.js + Go
AWS Infrastructure
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
When should
you scale?
Scaling Considerations
Hourly billing
Instance size: Cost vs Speed
Content priority
Ad-hoc requests
Varied content length
Scaling Considerations:
AWS limitations
AWS scaling metrics can be slow
No automatic off switch
Can’t choose instances to ...
MVP:
Don’t make things wait
too long
MVP:
Supply enough
capacity at all times
MVP:
Driven by drop folders
GXF, ABC1
Profile:
- Renditions (2)
- Crop
- Normalise audio
- Adjust ratio
- Add watermark
- GXF to MP4
var autoscaling = new AWS.AutoScaling(...);
var params = {
AutoScalingGroupName: groupName,
DesiredCapacity: capacity
Some...
var autoscaling = new AWS.AutoScaling(...);
var params = {
AutoScalingGroupName: groupName,
DesiredCapacity: capacity
Some...
Amazon Machine
Images (AMIs)
Which instance size?
Time vs Cost
for 1 hour of content
Time Cost
?
Winner:
c4.xlarge
Up: step scaling
Down: stop everything
An interesting puzzle :)
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
24 hours in
Transcoding
(10am - 10am)
$57.22
$42.86
US Dollars
EC2 Instances: $34.31
EC2 (other): $ 4.24
RDS: $ 2.66
S3: $ 1.65
SQS: $ 0.00
Total (USD): $42.86
Transcoding costs: 24 hrs
Next 24 hours
Approx 70% of the volume
$34.63
Australian Dollars
One year’s costs?
$16.5k
Australian Dollars
$12.5k
US Dollars
This is a BIG deal!
thesupercarkids.com
bikesandbeyond.ca
8 minutes 27 seconds
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
AWS & throughput
optimisations
Transcoding in
chunks/segments
Introspection &
Validation
GXF, ABC1
Profile:
- Crop
- Normalise audio
- Adjust ratio
- Add watermark
- GXF to MP4
ffprobe
1. What is Transcoding?
2. Why build our own system?
3. Architecture
4. Technology choices
5. Autoscaling transcoders
6. C...
Summary
8 minutes 27 seconds
@daphnechong
Thanks!
Q&A
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Upcoming SlideShare
Loading in …5
×

of

Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 1 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 2 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 3 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 4 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 5 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 6 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 7 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 8 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 9 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 10 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 11 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 12 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 13 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 14 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 15 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 16 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 17 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 18 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 19 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 20 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 21 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 22 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 23 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 24 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 25 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 26 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 27 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 28 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 29 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 30 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 31 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 32 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 33 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 34 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 35 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 36 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 37 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 38 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 39 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 40 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 41 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 42 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 43 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 44 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 45 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 46 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 47 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 48 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 49 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 50 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 51 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 52 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 53 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 54 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 55 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 56 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 57 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 58 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 59 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 60 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 61 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 62 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 63 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 64 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 65 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 66 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 67 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 68 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 69 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 70 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 71 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 72 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 73 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 74 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 75 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 76 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 77 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 78 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 79 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 80 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 81 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 82 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 83 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 84 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 85 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 86 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 87 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 88 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 89 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 90 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 91 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 92 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 93 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 94 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 95 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 96 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 97 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 98 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 99 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 100 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 101 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 102 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 103 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 104 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 105 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 106 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 107 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 108 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 109 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 110 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 111 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 112 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 113 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 114 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 115 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 116 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 117 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 118 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 119 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 120 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 121 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 122 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 123 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 124 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 125 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 126 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 127 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 128 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 129 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 130 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 131 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 132 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 133 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 134 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 135 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 136 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 137 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 138 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 139 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 140 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 141 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 142 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 143 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 144 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 145 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 146 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 147 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 148 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 149 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 150 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 151 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 152 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 153 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 154 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 155 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 156 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 157 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 158 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 159 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 160 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 161 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 162 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 163 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 164 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 165 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 166 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 167 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 168 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 169 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 170 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 171 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 172 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 173 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 174 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 175 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 176 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 177 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 178 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 179 Video Transcoding at Scale for ABC iview (NDC Sydney) Slide 180
Upcoming SlideShare
Video transcoding transrating
Next

3 Likes

Share

Video Transcoding at Scale for ABC iview (NDC Sydney)

ABC iview is a video on demand service from the Australian Broadcasting Corporation. In 2015, we built a new service in-house to handle transcoding for all of iview's content.

Metro is built on AWS, and uses FFmpeg, node.js and go components to transcode content quickly and cost-efficiently.

This talk was presented on August 4th 2016 at NDC Sydney.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Video Transcoding at Scale for ABC iview (NDC Sydney)

  1. 1. Video Transcoding at Scale for ABC iview Daphne Chong @daphnechong
  2. 2. metro
  3. 3. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  4. 4. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  5. 5. What changes are we making?
  6. 6. High speed Medium speed Low speed
  7. 7. Standard Encoding H.264 MP4
  8. 8. Transcoding Example
  9. 9. Step 1: Prepare watermark
  10. 10. Step 1: Prepare watermark
  11. 11. Step 1: Prepare watermark
  12. 12. Step 2: Input video (GXF)
  13. 13. Step 3: Asymmetrical cropping
  14. 14. Step 4: Scale and set aspect ratio
  15. 15. Step 5: Apply watermark
  16. 16. Step 6: Ready to stream 1500k
  17. 17. other rendition sizes 1000k 650k 500k 220k audio only
  18. 18. Original file audio track
  19. 19. Previous transcoder audio track
  20. 20. After audio normalisation
  21. 21. That’s a lot of changes...
  22. 22. Transcoding Profiles
  23. 23. GXF, ABC1 Profile: - Renditions (6) - Crop - Normalise audio - Adjust ratio - Add watermark - GXF to MP4
  24. 24. GXF, ABC1
  25. 25. GXF, ABC1 MOV, Arts Profile: - Renditions (6) - Adjust ratio - ProRes MOV to MP4
  26. 26. GXF, ABC1 MOV, Arts
  27. 27. What is this “Magic Video Software” you speak of?
  28. 28. ./ffmpeg -i utopia.gxf utopia.mp4
  29. 29. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  30. 30. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  31. 31. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  32. 32. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  33. 33. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  34. 34. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  35. 35. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  36. 36. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  37. 37. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  38. 38. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  39. 39. ./ffmpeg -i INPUT -c:v libx264 -filter_complex "[0:v]crop=720:576:0:32,yadif,scale=1024x576,setdar=16/9[ma in], [1:v]scale=54:-1[ovrl], [main][ovrl]overlay=main_w-overlay_w-47:main_h-overlay_h-44 " -crf 22 -maxrate 1404k -bufsize 1500k -pix_fmt yuv420p -profile:v high -level 4.0 -refs 3 -r 25 -g 50 -c:a libfdk_aac -b:a 96k -filter_complex "[0:a:0][0:a:1] amerge=inputs=2, dynaudnorm=f=500:g=31:p=0.82:m=30.0:r=0.0:s=0.0 [aout]" -ar 44100 -ac 2 -map 0:0 -map "[aout]" -movflags +faststart -threads 0 -y OUTPUT In reality, it looks like this
  40. 40. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  41. 41. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  42. 42. Cost Scale Flexibility
  43. 43. Cost Scale Flexibility
  44. 44. thesupercarkids.com
  45. 45. mini-me.com
  46. 46. Cost Scale Flexibility
  47. 47. “The Cloud”
  48. 48. Cost Scale Flexibility
  49. 49. Cost Scale Flexibility
  50. 50. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  51. 51. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  52. 52. Source Queue Process Transfer Notify
  53. 53. 8 minutes 27 seconds
  54. 54. Original length Time to produce rendition 60 min 7 min 42 min Lowest (audio) Highest (1500k)
  55. 55. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  56. 56. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  57. 57. Languages: Node.js Go
  58. 58. Node.js Existing team experience Rapid development for APIs Untyped. Sadface.
  59. 59. Go Typed and compiled Cross-platform Easy deployment Learning curve
  60. 60. Infrastructure: EC2, SQS, S3, SNS, Dynamo / RDS, Cloudformation
  61. 61. AWS - EC2 Autoscaling
  62. 62. AWS - SQS Simple Queuing System
  63. 63. SQS First in first out (FIFO) Single consumer per message Message has “visibility timeout” Adjustable visibility timeout
  64. 64. AWS - S3 File Storage
  65. 65. AWS - DynamoDB
  66. 66. AWS - DynamoDB
  67. 67. AWS - DynamoDB AWS - MySQL on RDS
  68. 68. DynamoDB NoSQL Key-value pair storage Pay for number of records/sec read
  69. 69. DynamoDB Our use case was not optimal Manual aggregations Higher cost than expected Doesn’t scale automatically
  70. 70. MySQL Switched to MySQL on RDS Managed hosting Aggregations! Data structure ported as-is
  71. 71. AWS Cloudformation
  72. 72. "ExampleLoadBalancer" : { "Type" : "AWS::ElasticLoadBalancing::LoadBalancer", "Properties" : { "AvailabilityZones" : [ "us-east-1a", "us-east-1b" ], "Listeners" : [ { "LoadBalancerPort" : "80", "InstancePort" : "8081", "Protocol" : "HTTP" } ] } }
  73. 73. Node.js + Go AWS Infrastructure
  74. 74. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  75. 75. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  76. 76. When should you scale?
  77. 77. Scaling Considerations Hourly billing Instance size: Cost vs Speed Content priority Ad-hoc requests Varied content length
  78. 78. Scaling Considerations: AWS limitations AWS scaling metrics can be slow No automatic off switch Can’t choose instances to scale down (Instance protection now available)
  79. 79. MVP: Don’t make things wait too long
  80. 80. MVP: Supply enough capacity at all times
  81. 81. MVP: Driven by drop folders
  82. 82. GXF, ABC1 Profile: - Renditions (2) - Crop - Normalise audio - Adjust ratio - Add watermark - GXF to MP4
  83. 83. var autoscaling = new AWS.AutoScaling(...); var params = { AutoScalingGroupName: groupName, DesiredCapacity: capacity Some white text }; autoscaling.setDesiredCapacity(params, ...);
  84. 84. var autoscaling = new AWS.AutoScaling(...); var params = { AutoScalingGroupName: groupName, DesiredCapacity: capacity Some white text }; autoscaling.setDesiredCapacity(params, ...);
  85. 85. Amazon Machine Images (AMIs)
  86. 86. Which instance size?
  87. 87. Time vs Cost for 1 hour of content Time Cost ?
  88. 88. Winner: c4.xlarge
  89. 89. Up: step scaling Down: stop everything
  90. 90. An interesting puzzle :)
  91. 91. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  92. 92. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  93. 93. 24 hours in Transcoding
  94. 94. (10am - 10am)
  95. 95. $57.22
  96. 96. $42.86 US Dollars
  97. 97. EC2 Instances: $34.31 EC2 (other): $ 4.24 RDS: $ 2.66 S3: $ 1.65 SQS: $ 0.00 Total (USD): $42.86 Transcoding costs: 24 hrs
  98. 98. Next 24 hours Approx 70% of the volume
  99. 99. $34.63 Australian Dollars
  100. 100. One year’s costs?
  101. 101. $16.5k Australian Dollars
  102. 102. $12.5k US Dollars
  103. 103. This is a BIG deal!
  104. 104. thesupercarkids.com
  105. 105. bikesandbeyond.ca
  106. 106. 8 minutes 27 seconds
  107. 107. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  108. 108. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  109. 109. AWS & throughput optimisations
  110. 110. Transcoding in chunks/segments
  111. 111. Introspection & Validation
  112. 112. GXF, ABC1 Profile: - Crop - Normalise audio - Adjust ratio - Add watermark - GXF to MP4
  113. 113. ffprobe
  114. 114. 1. What is Transcoding? 2. Why build our own system? 3. Architecture 4. Technology choices 5. Autoscaling transcoders 6. Costs 7. Future roadmap
  115. 115. Summary
  116. 116. 8 minutes 27 seconds
  117. 117. @daphnechong Thanks!
  118. 118. Q&A
  • MohdJounKazmi

    Nov. 23, 2020
  • ssuser51058b

    Jun. 1, 2017
  • SimonScott31

    May. 4, 2017

ABC iview is a video on demand service from the Australian Broadcasting Corporation. In 2015, we built a new service in-house to handle transcoding for all of iview's content. Metro is built on AWS, and uses FFmpeg, node.js and go components to transcode content quickly and cost-efficiently. This talk was presented on August 4th 2016 at NDC Sydney.

Views

Total views

4,097

On Slideshare

0

From embeds

0

Number of embeds

2,954

Actions

Downloads

0

Shares

0

Comments

0

Likes

3

×