This document discusses using Ceph and its RGW storage gateway to store live video streams. It outlines why Ceph/RGW was chosen, including its distribution, erasure coding, and CDN friendliness. It then describes the design and implementation of the live video system, including an API gateway, Nginx RTMP server integrated with RGW via librgw, and handling authorization. Some problems encountered are also mentioned.

1. RGW beyond Cloud
Shengjing Zhu, UMCloud
Yiming Xie, UMCloud
Live Video Storage with Ceph

2. Outline
➔ Background
➔ Why we choose Ceph/RGW to store videos
➔ How we design and implement the live video
system
➔ The problems we meet
➔ Demo & use cases

3. Background

4. Background
➢ Live streaming is a growing industry
➢ Video monitor: real-time analyse with AI
➢ Videos need a more powerful cloud storage
backend

5. Why RGW
➢ Ceph powered
○ Distribution
○ Erasure code
➢ A friendly API to access
○ Big data analyse over RGW
○ Machine learning
➢ CDN friendly
○ For HLS streaming

6. Design & Implement
➢ Architecture overview
○ An API gateway that fully integrates AWS
v2/v4 Authorization
○ A high performance Nginx RTMP server
integrates RGW with librgw

7. API Gateway
Client
API Server
Ceph RGW
http://api/?proxy_auth_headers http://s3?real_auth_headers
FFMPEG
Nginx RTMP
Server
rtmp://rtmp/?proxy_auth_heade
rs
http://api_notify/
?proxy_auth_headers

8. API Gateway - Authorization proxy
➢ HTTP endpoint
○ Pass through HTTP auth headers
➢ RTMP endpoint
○ When RTMP client "connect" server, the
auth headers can be set in "tcUrl"

9. Nginx RTMP
➢ On top of famous nginx-rtmp-module
➢ Support HLS encoding
➢ Store video in RGW
➢ Easy intergration with API gateway

10. Nginx RTMP - RGW
➢ S3Fuse
➢ NFS-Ganesha
➢ Raw HTTP access RGW
➢ librgw ✔
Pros & Cons ...

11. Nginx RTMP - Dataflow
Nginx Ceph
librgw
RGW
Dataflow path

12. Nginx RTMP - Worker
Nginx - master
librgw
instance
librgw
instance
librgw
instance
nginx
worker
nginx
worker
nginx
worker
Ceph Cluster
RTMP Client
RTMP Client

13. Nginx RTMP - Impl Detail
➢ Nginx worker process creates librgw
instance
➢ Replace API of HLS storage with librgw
➢ Add ACL control in librgw
➢ Set API gateway endpoint as notification
callback

14. Problems
➢ Concurrent
➢ librgw lacks of features like: ACL
➢ rename is slow
➢ AWS v4 authorization tokens are too long
➢ ...

15. Demo & User case

16. Thanks!
Q&A