This session will cover the approaches for a cloud-based workflow: media ingest, storage, processing and delivery scenarios on the AWS cloud. We will cover solutions for high speed file transfer, cloud-based transcoding, tiered storage, content processing, application deployment and global low-latency delivery, as well as the orchestration and management of the entire media workflow.
2. Media Workflows are Complex
Featurettes
Interviews
2D Movie
3D Movie
Archive
Materials
Stills
Archive
Networks
Theatrical
DVD/BD
Online
MSOs
Mobile Apps
Process
Store
Deliver
15. EBS Options:
• Three types:
– General Purpose (SSD) volumes
• Ideal for a broad range of use cases such as boot volumes, small and
medium size databases, and development and test
– Provisioned IOPS (SSD) volumes
• You specify the I/O performance, up to 4000 IOPS per volume
– Magnetic volumes
• Ideal for workloads where data is accessed infrequently, and
applications where the lowest storage cost is important.
• Encrypted Amazon EBS volumes to meet a wide
range of data-at-rest encryption requirements
Block (CIFS/NFS)
EBS
Elastic Block
Storage
16. Reducing Object Storage Costs
<$0.03/GB
Amazon S3
<$0.01/GB
Amazon Glacier
Single API access
(regardless of the
storage class)
> Durability
>> Scale
<$0.024/GB
>>> Cost Amazon S3 (RRS)
17. S3 & Working with Large Files
S3
Bucket
S3 SDK
AWS S3
S3 multi-part
Upload
Files
corporate data center S3 Object <= 5TB
18. S3 Encryption Options
1. Sever-Side Encryption (SSE)
– Provides an integrated solution where Amazon handles key
management
2. SSE-Customer (SSE-C)
– Perform the encryption and decryption of your objects while retaining
control of the keys used to encrypt objects.
3. Client library (e.g. Amazon S3 Encryption Client)
– Objects encrypted before they are sent to Amazon S3 for storage.
27. On-demand instances
Unix/Linux instances start at
$0.02/hour
Pay as you go for compute power
Low cost and flexibility
Pay only for what you use, no up-front
commitments or long-term contracts
Use Cases:
Applications with short term, spiky, or
unpredictable workloads;
Application development or testing
Reserved instances
1- or 3-year terms
Pay low up-front fee, receive significant hourly
discount
Low Cost / Predictability
Helps ensure compute capacity is available
when needed
Use Cases:
Applications with steady state or predictable
usage
Applications that require reserved capacity,
including disaster recovery
Spot instances
Bid on unused EC2 capacity
Spot Price based on supply/demand,
determined automatically
Cost / Large Scale, dynamic workload handling
Use Cases:
Applications with flexible start and end times
Applications only feasible at very low compute
prices
EC2 Instance Options
28. Managed Database Services
Amazon
RDS
Amazon
DynamoDB
Managed relational
database service
Managed NoSQL
database service
Amazon
ElastiCache
In-Memory Caching
Service
29. Workflow Building Blocks
Amazon Simple
Queue Service
(SQS)
Amazon
SWF
Amazon Simple
Email Service
(SES)
Amazon Simple
Notification
Service (SNS)
Amazon
CloudSearch
CloudWatch
AWS
CloudTrail
Amazon
ElastiCache
Amazon
Elastic
MapReduce
Elastic
Load
Balancing
Amazon
Relational
Database
Store
Deliver
Service (RDS) Process
30. AWS Media Delivery
Processed
Media Files HTTP/HTTPS
Amazon S3
Streaming or
Distribution
Amazon
CloudFront
37. Video Factory – Workflow
SDI Broadcast
Video Feed
x 24
Broadcast
Encoder
Live Ingest
Logic
Playout
Data Feed
Amazon Elastic
Transcoder
Elemental
Cloud
DRM
QC
Editorial
Clipping
MAM
Amazon S3
Mezzanine
Time Addressable
Media Store
Amazon S3
Distribution
Renditions
RTP
Chunker
Transcode
Abstraction
Layer
SMPTE
Timecode
Mezzanine Video Capture
Mezzanine
Playout Video
Transcoded Video
Metadata
38. • The UK’s biggest video & audio on-demand service
– And it’s free!
• Over 7 million requests every day
– ~2% of overall consumption of BBC output
• Over 500 unique hours of content every week
– Available immediately after broadcast, for at least 7 days
• Available on over 1000 devices including
– PC, iOS, Android, Windows Phone, Smart TVs, Cable Boxes…
• Both streaming and download (iOS, Android, PC)
• > 20 million app downloads
Sources:
BBC iPlayer Performance Pack August 2013
http://www.bbc.co.uk/blogs/internet/posts/Video-Factory
40. FIFA WORLD CUP
2014
June 12 - July 13 2014
- 12 Stadiums
- 64 Games
IBC1 in Rio de Janeiro
EVS appointed for multilateral production
- On site2 live production
- IBC file-based video management
- Multimedia production and distribution
EVS
SPORTS
Notes:
1. The International Broadcast Centre (IBC) is a temporary hub for broadcasters during major sport events.
2. Remote broadcast television studio where the event is produced and then beamed to broadcasters via
satellite.
41. EVS ENABLED A NEVER-SEEN-BEFORE MULTIMEDIA EXPERIENCE
6 Live Streams
HLS streaming of 6 HD streams
to tablets & mobiles per match
24 Replay cameras
On demand replay of selected events
from any of the 20+ cameras on the field.
+4000 VoD elements
On demand multimedia
exclusive edits.
44. Max of 12 streams
at 10Mbps each.
Max of 36 streams
at 3,5-10Mbps
9 HLS encoding, from
0.4 to 3.5Mbps + 2
thumbs
1 RTMP
EVS C-CAST LIVE STREAMING
WORKFLOW
45. EVS C-CAST VOD
WORKFLOW
9 HLS encoding, from
0.4 to 3.5Mbps + 2
thumbs
1 RTMP
46. EVS C-CAST MANAGES ALL FROM PRODUCTION TO VIEWERS
Cloud Platform
Linking mobile trucks, studios and viewers
Content selection and approval process
Metadata editing station
Automatic translations in 20+ languages
Automatic HD backup on top
Workflow engine, including
- VoD file transfers and transcoding management
- Live streaming activation and transcoding management
Output: API mode or white label embedded
EVS
SPORTS
48. MULTIMEDIA PRODUCTION
IN THE AWS CLOUD
Aspera for transfers into S3
Store and deliver data using S3
Create image thumbnails (EC2, SQS & S3)
Transcode VoD files using Zencoder
Transcode Live using Elemental Cloud
- Up to 128 concurrent live streams encoded
Operator GUI for Rio and international sites
- S3, Cloudfront, EC2 and RDS
Manage all workflows at IBC and Venue
- EC2, SQS & RDS
1Gbps Direct Connect from Rio to Dublin
49. EVS
SPORTS
6 MULTILATERAL STREAMS
17 INDIVIDUAL MRL STREAM
4 INDIVIDUAL COMMENTARY
243 DIFFERENT LIVE STREAMS /GAME
2,799,360 MINUTES OF ENCODED STREAMS
LIVE STREAMING
50. MULTI-ANGLE CLIPS
On average 60 CLIPS RECORDED PER MATCH, overall
UP TO 20,000 MULTILATERAL CLIPS throughout the tournament
Roughly a third of clips from multicam with 14 DIFFERENT ANGLES on
average
51. COMBINED DATA FOR FIFA WORLD CUP WEB PLAYER
AND APP AFTER 64 MATCHES FOR ALL MARKETS
TOTAL HOURS WATCHED:
>15 MILLION
UNIQUE USERS:
>35 MILLION
Available in more than 10
TERRITORIES
Peak ACTIVE CONCURRENT USERS
500,000
52. FIN & THANK YOU!
Hall #3.B16
Hall #7.G30
Hall #4.B75
Hall #8.B90
Hall #5.B20
Editor's Notes
Start by talking about media workflows. Main point is there are many workflows.
Use media workflows to go from what’s on the the left to what’s on the right.
Steps themselves are generally pretty straightforward.
Industry trends that are making workflows more complex:
More content: at the pro end look at all the content on the left. On the consumer end, everyone is carrying around a 1080p camcorder. And the more content there is, the greater the opportunity to monetize it.
Bigger content: the industry is moving to some combination of more pixels, faster pixels and better pixels. More pixels: 4K and beyond (4x pixels compared to 1080p). Faster pixels: higher frame rates. 48fps is 2x current cinema frame rate. Better pixels: higher dynamic range and brighter pixels, increase bit-depth.
More processing: the amount of processing going up not down. At the high end, whether it is a commercial, a TV show or a movie, most shows contain visual effects. Even in corporate video, color correction is becoming a standard part of the workflow. And at the consumer level, all those Instagram like filters require processing.
More output formats: not just renditions based on devices but also versions. ? One senior industry figure recently told me that a piece of finished content will have been converted 1000 times!
So all of these trends have an impact on workflows especially when you factor in constrained budgets and timeframes.
To give you context for what follows in Phil’s session, I thought I’d cover where AWS fits and then some approaches we’ve seen for doing media processing at scale in the cloud.
As you know, AWS provides infrastructure services: compute, networking, database, storage and delivery and so on. We also provide application services and deployment and management services. Using these services, as your “software defined datacenter”, you can build media processing workflows.
Typical operations in a media workflow would run on top of the AWS services. These operations could be provided by software that you’ve developed or they might be from another vendor like Aspera for ingest or Tektronix for video QC.
On top of all that you’d have media applications – perhaps an Online Video Platform, a production management application, a digital dailies system or visual effects.
So that’s where AWS fits. Now let’s look at some approaches for doing media processing on AWS.
To give you context for what follows in Phil’s session, I thought I’d cover where AWS fits and then some approaches we’ve seen for doing media processing at scale in the cloud.
As you know, AWS provides infrastructure services: compute, networking, database, storage and delivery and so on. We also provide application services and deployment and management services. Using these services, as your “software defined datacenter”, you can build media processing workflows.
Typical operations in a media workflow would run on top of the AWS services. These operations could be provided by software that you’ve developed or they might be from another vendor like Aspera for ingest or Tektronix for video QC.
On top of all that you’d have media applications – perhaps an Online Video Platform, a production management application, a digital dailies system or visual effects.
So that’s where AWS fits. Now let’s look at some approaches for doing media processing on AWS.
[Mention that Sony MCS went through the full MPAA audit]
Amazon S3 Reduced Redundancy
99.99% durability vs. 99.999999999%
Up to 20% savings
Great for everything that is easy to reproduce
Amazon Glacier
Same durability as S3
3 to 5 hours restore time
Up to 89% savings
Great for archiving, long-term backups and old data
Amazon EC2 offers a broad selection of virtual server configurations, or instance types, to satisfy the resource needs of a diversity of workloads. And we have been iterating quickly on behalf of our customers. These can be purchased on-demand, meaning you only pay for the hours of time that you use them. This has the huge advantage of turning capital expenditures into variable costs, one reason why AWS has been so popular amongst small businesses and enterprises. And when you look as our instance types, each has a different focus – memory, cpu, storage, IOPS, and GPUs each of which align with different use cases. We got into the GPU family 3 years ago to provide customers on-demand access to the massive parallel processing capability of GPUs. For workloads that fit into the GPU co-processor programming model, the performance boost can be huge.
After launching our first GPU instance – which supported GPU compute programming models only - we started getting customer feedback about the Cluster GPU instances. For some, they were too expensive. For others, they were overkill. As programming two GPUs was more cumbersome at the time. And the last and most prominent piece of feedback was the desire for graphics API. As an interesting side, some customers found ways to run low versions of OpenGL, providing us demonstrated demand for the graphics APIs. We knew we had to evolve the instance family.
Roughly four months ago we launched the G2 instances type, which we believe accommodates the customer feedback we’ve received. It has newer GPU hardware from NVIDIA, a single GPU per instance, supports graphics APIs in addition to GPU compute programming models, and it comes at a significantly reduced price of $0.65 per hour, Linux, on-demand in our Northern Virginia and Oregon datacenters.
Cost is the conversation starter when it comes to cloud. There are many pieces to cost conversation when it comes to AWS and your own infrastructure. The first advantage you get in the cloud is that you don’t have to lay out capital expense for hardware and infrastructure before you know the demand. In essence you convert your capital expense into variable expense. And then that variable expense on AWS is lower than what most companies can do on their own because AWS runs at a massive scale and we pass that scale to our customers in the form of lower pricing. There are multiple pricing models in AWS, so you can optimize your spend depending on what your workloads requirements are. And the more you use AWS, the less your costs are. We have tiered pricing and for customers doing large data center migrations, we have negotiated custom pricing to make their transitions cost-effective.
NOTES:
This architecture shows how to deliver content in popular formats including:
Apple’s HTTP Live Streaming (HLS)
Adobe’s HTTP Dynamic Streaming (HDS)
Microsoft’s Smooth Streaming
This diagram doesn’t explicitly call out the upcoming MPEG-DASH format, but it would be very similar since it is based on the 3 formats shown
Transcoding is outside the scope of this diagram. There are many transcoding options available
For live streaming, the transcoder will push the content to the segmenters via different protocols depending on the type of ABR streaming:
For HLS, MPEG2 Transport Streams are used to deliver to the Apple Stream Segmenter tool
For HDS, RTMP is used to deliver to the Adobe Live Packager tool
For Smooth Streaming, the Smooth Streaming Live Protocol is used
After encoding, but prior to packaging, you may also choose to encrypt the content using one of many popular DRM technologies, which are also specific to the various ABR technologies:
HDS uses Adobe Access or Protected HDS
HLS uses HLS encryption
Smooth Streaming uses PlayReady
It should also be noted that both Adobe and Microsoft added the ability to deliver HLS formats to Adobe Media Server and IIS Media Services, respectively
[Mention that Sony MCS went through the full MPAA audit]
Introduction:
Phil Cluff
Principal Software Engineer & Team Lead @ BBC Media Services
Been with BBC for 3 ½ years, focused in Transcode architectures, Message Orientated Middleware & Reliable, Distributed systems in the cloud!
I’m going to talk to you about BBC iPlayer and our journey into the Cloud.
Be really clear on Mezzanine definition since next 4 slides depend on it.
Mention Mez video capture is classic broadcast technologies.
Make note of the “Time addressable media store”
Hopefully you’ve all heard of the BBC, but you may not have all heard of iPlayer.
So What is BBC iPlayer?
UK online population is about 40m which is the size of the state of California.