1. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
조휘열본부장
콘텐츠연합플랫폼(주)
국내 미디어 고객사의 AWS
활용 사례 - POOQ

2. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Agenda
Brief History
Genesis
POOQ 1.0
POOQ 2.0
Head End
POOQ 3.0

3. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
About: Me
• 조휘열
• 콘텐츠연합플랫폼 플랫폼기술본부장
• 플랫폼 설계/개발/운영
+ 클라이언트 설계/개발/운영
• 개발 경력 29년
felix@captv.co.kr

4. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Brief History

5. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
About: POOQ

6. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
About: POOQ
• JV: KBS, MBC, SBS
• 64 Live Broadcasting stream
• Over 170,000 VOD streaming
• About 7,000 Movie streaming
• About 660,000 Paid subscriber
• About 3,400,000 Registered users

7. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
History of POOQ
2012 2013 2014 2015 2016 2017
2012년 5월
회사설립
2012년 7월
POOQ 1.0
2014년 2월
로컬 클라우드
2015년 6월
POOQ 2.0
2015년 11월
프리미어12
2018
2017년 2월
AWS
2018년 4월
POOQ 3.0
2016년 10월
Head End

8. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Genesis

9. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Genesis of POOQ
• Initial platform comes from MBC’s
• iMBC had own service called “POOQ”
• Transfer IP & system to CAP
• Microsoft .Net based
• Engineer from SBSCH joins CAP

10. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Genesis of POOQ (Con’t)
• SBSCH incorporate Java based technology
• Quite unique technical issues from the start
• Two different development languages exists
=> Polyglot became mandatory
• Had to utilize existing platform

11. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
POOQ 1.0

12. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
POOQ 1.0
• Most of platform comes from iMBC
• Hosted at Nonhyun
IDC (LG)
• Typical Web
App Structure
ASP.Net
(PC Web
Service)
MS SQL
(Data
Storage)
Ingestion
Engine
MAPI
(Mobile API
Server)
ASP.Net
(Backoffice)

13. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Scalability problem
• With number of users grow fast, problem start rise
• Occasional heavy load – could not stand over 80,000
concurrent users
• Required server reset
• Poor user experience

14. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Evidence 2013/10/19

15. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Evidence 2013/10/20

16. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Evidence 2013/11/8

17. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Evidence 2014/4/24

18. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
POOQ 2.0

19. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Goal
• Die Hard
• Scalable Platform -> Cloud
• Unification
• Unified API for all supported devices
• Personalization
• My view
• Recommendation

20. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Die Hard – Cloud
• Two options:
• Local cloud
• AWS Tokyo
• Regulatory issue:
• “One cannot move customer’s personal record outside of
country without consent”
• Destination: Local cloud
• Problem was, we really did not know about local ucloud
• No good document or books exist as well
• Though cloud is almost same

21. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Die Hard – Scalability
• Fully Scalable
MongoDB
(Slave)
Web Servers
(Static Page)
LB Group
Web Servers
(Static Page)
Web Servers
(Static Page)
Web Servers
(Static Page)
Web Servers
(Static Page)
LB Group
Web Servers
(Static Page)
Web Servers
(Static Page)API Servers
MongoDB
(Slave)
MongoDB
(Master)
Cassandra
Ring
Node
Node
Node
Node
Node
Node
Web Servers
(Static Page)
LB Group
Web Servers
(Static Page)
Web Servers
(Static Page)
Bookmark
Collectors

22. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Unification
• Eliminate duplicated resource allocation
• Same behavior on all client
ASP.Net
MAPI
Unified API

23. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Unification
• Swagger (http://swagger.io/)
• Restful API definition with .yaml
• Free documentation, testing tools available
• Auto-generate client & server for several languages

24. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Unification
• https://api35-docs.pooq.co.kr/

25. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Personalization
• Personalization
• Continues viewing – Across N-Screen
• Popular content listing
• Your program listing
• All based on “Bookmark” concept
• Client send streaming status to server
for every 10 second
• HTTP based protocol support
• Heavy load on server side

26. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Personalization – Continues viewing
• 10 second resolution
PC
Mobile
Viewing
Viewing
Viewing

27. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Personalization – Continues viewing
• 8TB/Month Bookmark
Receiver
Bookmark
ReceiverBookmark
Receiver
Client ActiveMQ
Bookmark
Receiver
Bookmark
Cassandra
Data Store
API Server
Apache SparkMongo DB
Service

28. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Tools
• Kettle ETL tool
• Apache Jmeter performance measure tool
• Github
• Youtrack
• Jenkins

29. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Overall architecture
Browser
Client
iOS
Client
Android
Client
Smart TV
Client
Web Servers
(Static Page)
Member Web
Servers
API Servers
Bookmark
Collection
Servers
Edge Servers
Product &
Rights Servers
Contents
MongoDB
Member
MongoDB
SysConfig
MongoDB
External
Interfaces
MS SQL
Backend
PentahoKettleETL
CP Metadata
Ingestion
CP Media
Ingestion
Origin Servers StorageCDN
Cassandra
NoSQL
Backend
Clusters

30. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Result?
• POOQ obtain exclusive Internet streaming rights on
Premier12 Baseball Game
• 19th November, 2015 - Premier12, Korea vs. Japan

31. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Result (Con’t)
• 551G Network
• 283,577 Users
• Platform was stable
• CPU Utilization was 20%

32. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Head End

33. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Head End
• Responsible for media preparation & streaming
• Project started 3Q, 2016
• First platform hosted on AWS
• Business and functional requirement to have own head
end rises
• Desire for better service quality and diversification
• UHD/4K readiness

34. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Head End (Con’t)
• Supported protocol: HLS, MPEG-DASH
• Supported codec: AVC(H.264), HEVC(H.265)
• Supported DRM: Widevine Classic, Widevine Module,
Play Ready, Fair Play
• Support VBR streaming

35. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Live Streaming
• With AWS Elemental Live encoder
• Handles 64 Live streaming including audio-only
channels
• Supporting: Live HLS, MPEG-DASH
Client
AWS
Elemental
Live
External Live
Feed
(RTMP)
AWS Live
S3 bucket
Protocol
Muxer
Live Proxy,
Manifest
Generator
AWS Live
Streaming
CF
Live
Thumbnail
Generator
AWS Live
Thumbnail
CF
External
Radio Feed
(RTMP)
WowzaRadio DVR

36. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
VOD Streaming
• With proprietary VOD Ingest Worker (Based on FFMPEG)
• Session-based watermarking technology
Client
VOD
Ingest
Worker
TV VOD
(HTTP Pull)
AWS
Vod S3
bucket
AWS VOD
Streaming CF
Movie
Watermarkin
g Embedder
Movie
VOD
(FTP Push)
Movie
Ingest
G/W
Signed URL
Generator

37. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
POOQ 3.0

38. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Objective
• Better scalability
• Massive API server -> Microservice
• Die harder
• Circuit breaker, Standardized local cache
• Less time to market
• Only need to test modified API with Microservice
• Better data analysis
• Neural Net

39. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Goal
• Docker based Microservice
• Increase development & deploy efficiency
• Auto scale on different level
• Polyglot development environment
• Circuit Breaker, Local Cache in framework
• Select 5 languages:
Java, C#, Javascript(Node.js), Python, Golang

40. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Goal (Con’t)
• Docker Orchestration:
• Kubernetes
• Open Source
• Operation Team’s favorite

41. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Goal (Con’t)
• API Gateway:
• Kong API Gateway
• Initially AWS API Gateway was selected
• Hit maximum of 50 LBs per VPC (Kubernetes with AWS API
Gateway case, solved by AWS now)
• Had to look for other way and select Kong

42. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Goal (Con’t)
• Message bus:
• Apache Kafka Message Broker
• Centralized, Standardized Messaging Mechanism
• Can store messages for long time (Default is 7 days)
• All user activity will store via Kafka (API log, Click log)

43. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Goal (Con’t)
• Marketing campaign engine
• Proprietary solution
• Monitor full user activity via Kafka for specific threshold
• Activate pre-designed campaign
(Like discounted subscription offer)
• From code based to rule based

44. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Goal (Con’t)
• Machine learning
• Scene keyword detection
• Celebrity face identification
• Contents related article analysis
• AWS Rekognition and Clarifai
• Google Tensorflow as in-house tool

45. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
POOQ 3.0 Overview
PC Browser
(Client
Rendering)
Meta
Ingestion
Mobile Client
POOQ Streaming H/E
Media
Ingestion
Data Sync
Smart TV
Client
AWSCF
Kubernetes
Kong
API
API
…
Member
MongoDB
Contents
MongoDB
User Behavior Data
Collector
ApacheKafka
MessageBroker
Member
MSSQL
Machine
Learning
Contents
MariaDB
Marketing
MongoDB
CassandraDB
DataLake
S3
Kettle ETL
Marketing
Engine
AWS EMR
Data Analysis
Marketing
Planner