Introduce how to build a media platform by using Microservices architecture on AWS.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
John Chang
Ecosystem Solutions Architect
April 2016
利用微服務架構建立
雲端影音平台

2. The Problem

3. Your Consumers Have Changed
More content choices
More devices
More delivery methods
New experiences

4. Your Business Has Changed
More data
Faster go to market
Leaner teams
Innovation and failure

5. Clarifications
OTT, OVP, AVOD, TVOD, SVOD
It doesn’t matter
Microservices
Buzzword?
Specific scope, interoperable services
Allow rapid innovation

6. Over the Top Platform

7. Content
Production
Content
Storage

8. Processing &
Management
Content
Production
Content
Storage

9. Processing &
Management
Content
Production
Content
Distribution
Content
Storage

10. Processing &
Management
Content
Production
Content
Distribution
Content
Storage
Content
Consumption

11. Processing &
Management
Shared IT Services
NetworkInfrastructure OperationsSecurity
Content
Production
Content
Distribution
Content
Storage
Content
Consumption

12. Processing &
Management
Content
Production
Content
Distribution
Content
Storage
Content
Consumption

13. Your Existing Physical Requirements
HD-­SDI
Satellite transmission
Local SAN
HW transcoder

14. Your OTT Approach
FTP / Accelerator
Large NAS
DAM / Workflow System

15. Storage and Ingest — Serverless
AWS Import/ Export
S3 Ingest
Content
provider

16. Storage and Ingest — Serverless
AWS Import/ Export
S3 Ingest
Lifecycle
policies
S3 bucket
Amazon Glacier
Content
provider
Lifecycle
policies
S3 IA

17. Storage and Ingest — Serverless
AWS Import/ Export
S3 Ingest
S3
notification
AWS Lambda
Lifecycle
policies
S3 bucket
Amazon Glacier
Content
provider
Lifecycle
policies
S3 IA

18. Storage and Ingest — Serverless
AWS Import/ Export
S3 Ingest
S3
notification
AWS Lambda
Metadata
extract & persist
Amazon
DynamoDB
Amazon
CloudSearch
S3 bucket
Amazon Glacier
Content
provider
Lifecycle
policies
Lifecycle
policies
S3 IA

19. Storage and Ingest — Serverless
AWS Import/ Export
S3 Ingest
For source, renditions, metadata sidecar files
S3
notification
Amazon Elastic
Transcoder
proxy / thumbnail
generation
AWS Lambda
Metadata
extract & persist
Amazon
DynamoDB
Amazon
CloudSearch
S3 bucket
Amazon Glacier
Content
provider
Lifecycle
policies
Lifecycle
policies
S3 IA

20. Storage and Ingest — Serverless
AWS Import/ Export
S3 Ingest
For source, renditions, metadata sidecar files
API Gateway
CloudFront
distribution
S3
notification
Amazon Elastic
Transcoder
proxy / thumbnail
generation
AWS Lambda
Metadata
extract & persist
Amazon
DynamoDB
Amazon
CloudSearch
AWS Lambda
S3 bucket
Amazon Glacier
Content
provider
Lifecycle
policies
Lifecycle
policies
S3 IA

21. Processing &
Management
Content
Production
Content
Distribution
Content
Storage
Content
Consumption

22. Encoding, Packaging, Encrypting
Challenges
Multiple client devices
Higher quality content
Parallel, complex workflow
Uneven load distribution

23. Content Processing Pipeline (Using Lambda)
AWS Import/ Export
S3 Ingest
Ingest

24. Content Processing Pipeline (Using Lambda)
AWS Import/ Export
S3 Ingest
S3 notification
Ingest

25. Content Processing Pipeline (Using Lambda)
AWS Import/ Export
S3 Ingest
S3 as backend storage for content files accessible
to other processing tasks
Amazon Elastic
Transcoder
S3 notification
Trigger a Lambda
function to start a
transcoding job
Ingest

26. Content Processing Pipeline (Using Lambda)
AWS Import/ Export
S3 Ingest
S3 as backend storage for content files accessible
to other processing tasks
Amazon Elastic
Transcoder
S3 notification
Trigger a Lambda
function to start a
transcoding job
Ingest
S3 notification

27. Content Processing Pipeline (Using Lambda)
AWS Import/ Export
S3 Ingest
S3 as backend storage for content files accessible
to other processing tasks
Amazon Elastic
Transcoder
S3 notification
Trigger a Lambda
function to start a
transcoding job
Ingest
S3 notification
Lambda function to
extract metadata
Update CMS or
metadata

28. Content Processing Pipeline (Using Lambda)
AWS Import/ Export
S3 Ingest
S3 as backend storage for content files accessible
to other processing tasks
Amazon Elastic
Transcoder
S3 notification
Trigger a Lambda
function to restart a
transcoding job
Ingest
S3 notification
Lambda function to
extract metadata
Update CMS or
metadata
API Gateway

29. Processing &
Management
Content
Production
Content
Distribution
Content
Storage
Content
Consumption

30. Live
RTMP ingest to origin
Repackaged and encrypted on
the fly
DRM less common
WebRTC for 1-­to-­few, then
RTMP/HTTP when traffic
increases
Content Delivery Architectures
VOD / Catchup
File-­based – stored on S3/CDN
Usually prepared
(encrypted/packaged) before final
storage
DRM by default
ISPs worried about unicast
delivery

31. Multi-­CDN

32. CDN Selection Methods
DNS-­based
Geo / latency / intelligent routing at DNS level
Asset sharding
50% of assets on CDN A, 50% on CDN B
CDN-­aware asset info service

33. CDN-­aware Asset Info Service
AWS Elastic
Beanstalk
CDN
Authentication
Authorisation
Asset info
request

34. CDN-­aware Asset Info Service
S3 Ingest
Ingest
Amazon
DynamoDB
Media processing
Amazon Elastic
Transcoder
Processing

35. CDN-­aware Asset Info Service
S3 Ingest
Ingest
AWS Elastic
Beanstalk
Amazon
DynamoDB
Media processing
Amazon Elastic
Transcoder
Processing
Authentication
Authorisation
Asset info
request

36. CDN-­aware Asset Info Service
S3 IngestAuthoritative Source – for multi-­CDN
Ingest
AWS Elastic
Beanstalk
CDN
Amazon
DynamoDB
Media processing
Amazon Elastic
Transcoder
Processing
Authentication
Authorisation
Asset info
request

37. S3 IngestAuthoritative Source – for multi-­CDN
Ingest
API
Gateway
CDN
Amazon
DynamoDB
Media processing
Amazon Elastic
Transcoder
Processing
Authentication
Authorisation
CDN-­aware Asset Info Service
Perfect for microservices – and for serverless computing
Lambda

38. Processing &
Management
Content
Production
Content
Distribution
Content
Storage
Content
Consumption

39. Subscriber Concurrency Management
Heartbeat sent from every
player at regular intervals
Supports both native and
web-­based players
Amazon
Kinesis
Lambda

40. Subscriber Concurrency Management
Heartbeat received by Amazon
Kinesis
Data fed into DynamoDB
DRM licence server reads
DynamoDB table
Decision made
Amazon
Kinesis
DynamoDB
Lambda
EC2

41. Subscriber Concurrency Management
Amazon
Kinesis
Lambda
Track drop-­off
Highlight popular content
Feed back into content
developmentDynamoDB Amazon Redshift

42. Subscriber Concurrency Management
Amazon
Kinesis
Lambda
Syndicate data to ads server
Recommendations-­based
demographic
DynamoDB Amazon Redshift
Amazon MLAd server

43. Subscriber Concurrency Management
Amazon
Kinesis
Lambda
Cross-­device
Play / pause / resume
No servers needed
Client-­side “stop” can be
enabled, too (non-­DRM based)
DynamoDB Amazon Redshift
Lambda
API Gateway

44. Best Practices
Avoid a priori optimization
Deploy often
Keep simple and separate systems
Don’t always give in to “right tool for the right job”

45. Summary
• Avoid a priori optimization
• Deploy often
• Keep simple and separate
systems
• Don’t always give in to “right
tool for the right job”
Processing &
Management
Content
Production
Content
Distribution
Content
Storage
Content
Consumption

46. Summary
• Serverless by default
• Avoid lock-­in
• Be cautious of the outside
• Reuse data for good
• Avoid a priori optimization
• Deploy often
• Keep simple and separate
systems
• Don’t always give in to “right
tool for the right job”
Processing &
Management
Content
Production
Content
Distribution
Content
Storage
Content
Consumption

47. Summary
• Serverless by default
• Avoid lock-­in
• Be cautious of the outside
• Reuse data for good
• Avoid a priori optimization
• Deploy often
• Keep simple and separate
systems
• Don’t always give in to “right
tool for the right job”
Go Build.

48. Thank you!