The ABC recently launched a new video transcoding system to process all the video content for ABC iview, our catch-up TV service.
Metro is a cost-efficient, scalable, cloud-based system that was built using Golang, Node, FFmpeg, and heavily utilises a variety of AWS technology including queues, varied capacity autoscaling, hosted database servers, and notifications. The system has been live since December 2015, and has successfully processed thousands of pieces of content.
78. metro - Media Transcoder
Node.js
- Existing team experience
- Rapid development
- Solid choice for APIs
- Reservations for long lived processes
- Untyped. Sadface.
87. metro - Media Transcoder
SQS
- First in first out (FIFO)
- Single consumer per message
- Message has “visibility timeout”
- Adjustable visibility timeout
95. metro - Media Transcoder
DynamoDB
- NoSQL
- Key-value pair storage
- Managed hosting
- Pay per record retrieval throughput
96. metro - Media Transcoder
DynamoDB
- Our use case was not optimal
- No aggregations = no stats
- Higher cost than expected
- Doesn’t scale automatically
97. metro - Media Transcoder
MySQL
- Switched to MySQL on RDS
- Managed hosting
- Aggregations!
- Data structure ported as-is
98. metro - Media Transcoder
Node + Go
- AWS: EC2, SQS,
- S3, Dynamo/RDS
99. metro - Media Transcoder
System Overview:
Autoscaling
Transcoders
102. metro - Media Transcoder
Scaling Considerations
- Hourly billing
- Instance size: Cost vs Speed
- Content priority
- Ad-hoc requests
- Varied content length
103. metro - Media Transcoder
AWS limitations
- AWS scaling metrics can be slow
- No automatic off switch
- Can’t choose instances to scale down
- (Instance protection now available)