1) ITV wanted to distribute large volumes of video content to partners in a scalable and cost-effective way without using a push model that led to high bandwidth costs. 2) They built a shared content distribution pipeline on AWS using S3 for storage, SNS for notifications, and Lambda functions. This allows content to be uploaded once to S3 and then partners can pull the content through their own AWS accounts. 3) Setting up the infrastructure as code with CloudFormation templates allows them to easily replicate test, staging, and production environments and make consistent infrastructure changes across environments.

1. Building a Shared Pipeline
for Content Distribution
AWS Media and Entertainment Cloud Symposium
23rd March 2016
Jim Taylor & Sam Bashton

2. Who are we? - ITV/Jim
● Jim Taylor - Technical Product Owner, VOD Distribution
● jim.taylor@itv.com
ITV is the biggest commercial television network
in the UK. We love telly, we’re passionate
about content and most importantly we
understand our audiences.
As a growing international company, we’re
commercially minded and have a global focus.
We’re all about pushing the boundaries and
being innovative.

3. Who are we? - Sam Bashton

4. VOD Distribution - Push model
ITV
Partner 1
Push content
Partner 2Push content
Scale issues with:
● Bandwidth
● Processing
● Unique delivery methods
● Redelivery requests
...but same content to each partner!
Partner 3
...

5. Business problem
How can we continuously distribute large volumes of high quality
video to several destinations, without spiralling distribution costs?
● Partners need high quality mezzanine so they can transcode to multi-platform
● ITV has about 200 hours/week of Catch Up ≈ 6TB/week
● 1Gbps link per partner = high cost/long lead time
● Push model unsustainable as distribution points increase

6. Solution - Cloud delivery - pull model
Use AWS to deliver once using a templated delivery and offer partners a fixed menu via a pull model
ITV AWSITV
Partner 1 AWS
Partner 1
Upload
once
Download
content
Notifications
Notifications
Partner 2
Partner 2 AWS
Notifications
Download
content
Notifications

7. Solution - Package
Shared:
Series/Episode Images 16:9 & 4:3
HD (XDCam50) or SD (IMX30) asset
Private:
Metadata file containing
● Tech metadata, asset locations
● Licence details
● Editorial metadata

8. Technology used
ITV AWS
ITV
Partner AWS
Partner
Signiant Flight
SNS
Amazon S3
AWS CloudFormation
Amazon Lambda
ELK stack on EC2
Splunk

9. Partnership - ITV & Bashton

10. Partner Setup: AWS
● Each partner has their own AWS account
● Files in the bucket use ‘requester pays’ billing - partners pay for data transferred
outside of AWS
● Partners notified via SNS when new content is available

11. Partner setup: SNS
● ‘Simple Notification Service’
● Push notifications
○ Webhooks
○ SQS (Simple Queue Service)
○ Lambda

12. Lambda
● Code without a server*
● Trigger simple tasks
● Used as part of shared pipeline
○ Receives notifications when videos are uploaded
○ Looks up partners, sends SNS notifications to those that are relevant

13. Test environments
● Creating resources via web console fine for trying things out
● Proof of concept environment created in this way
● In production, we need to build multiple copies of the environment
● Changing everything manually quickly becomes tiresome

14. CloudFormation
● CloudFormation to the rescue!
● Infrastructure as Code
● Specify what an environment looks like in a JSON text file
○ Networks, subnets
○ Firewall rules
○ S3 Buckets
○ Instances, volumes
○ Access control
● Commit to source control (GIT)

15. CloudFormation: Rich documentation
● Every change has a commit message
● History of every setting can be viewed back to the beginning of the project
○ With comments about every change to say why they were made

16. CloudFormation: Testing
● Change a template
● Apply to test environment
● Run the same change against staging, production
● Confidence that every change will work, and that all tests are valid

17. ● Lambda - code without servers*
● Code still needs to be tested
● For software deployed to instances we use Jenkins for CI
○ Automated Testing
○ Deployment and ‘promotion’ between environments (test->stage->prod)
● What to use with Lambda?
Software Testing

18. ● Jenkins
● Code committed to git repo
● Jenkins checks out code from git
● Runs automated tests
● Deploys to environment
● Deploys to ‘next’ environment (test->stage) at the click of a button
Jenkins + Lambda

19. Future plans
ITV AWSITV
Partner 1 AWS
Partner 1
Upload
once
Download
content
Notifications/
metadata
Notifications
Partner 2
Partner 2 AWS
Notifications
Download
content
Notifications/
metadata
Metadata API
Added ourselves as a Shared
Pipeline partner!
Add access services
Cloud transcoding & stitching

20. ● Deliver a ‘reference’ test system for partners - easier to look at a working example than documentation
● Issues using “requester pays” in egress - make this clearer
● Avoid logging overload
What would we do differently?

21. ● Delivery time of project - 5 months
● Reduced time to deliver a new partner from 3-6 months to 1-2 months
○ ...providing they use our system
● Reduced cost to deliver a new partner by up to 75%
○ ...providing they use our system
● Can start testing within a week
Success? - Yes!