2. Who Am I?
• 조휘열
• 콘텐츠연합플랫폼 플랫폼운영실장
• 서비스 기획/서버&클라이언트개발/플랫폼&서비스 운영 담당
• 개발 경력 26년
• felix@captv.co.kr
3. In the beginning
• May, 2012 - CAP(Contents Alliance Platform Co., Ltd.) Formed
Joint venture between MBC & SBS
• July, 2012 – POOQ service open to public
• September, 2012 – POOQ became paid subscription service
5. POOQ 1.0
• Most of platform comes from iMBC
• Hosted at Nonhyun IDC (LG)
• Typical Web App Structure
ASP.Net
(PC Web
Service)
MS SQL
(Data
Storage)
Ingestion
Engine
MAPI
(Mobile API
Server)
ASP.Net
(Backoffice)
6. Houston, we have problem
• With number of users grow fast, problem start rise
• Occasional heavy load – could not stand over 80,000 concurrent users
• Required server reset
• Poor user experience
11. Why?
• IDC
No rapid platform growth
Everything should planned month ahead
• Old technology
RDBMS backed, real-time query design
Developed cache on ASP.net but it was complex and not efficient
Cannot increase number of servers (of RDBMS)
Everybody hates “Sharding”
12. New Game Plan
• New type of service: “Emergency Platform”
RDBMS-less, Authentication-less
Operate when primary platform failed
• From IDC to Cloud
Rapid Scalability is the name of game
• New platform design
Unified API
Personalization
Scalable Performance
13. Emergency Platform
• Preparation for Primary Platform Failure
• Collect live & VOD information at booting
• Does not rely on any regular platform servers
• No authentication – When primary server failed, everybody can watch VOD!
ASP.Net
(PC Web
Service+MAPI
emulation)
External Data
Source
(live & VOD
info)
Emergency Platform
From IDC to Cloud
New Platform Design
14. From IDC to Cloud
• Immigration is decided, problem is to where?
• Two options were considered
KT ucloud
AWS Tokyo
• Regulatory issue makes a complication for AWS option
“One cannot move customer’s personal record outside of country without consent”
• KT ucloud was cheaper, too
• So, destination : KT ucloud
• Problem was, we really did not know about KT ucloud
No good document or books exist as well
Though cloud is almost same Big Mistake
Emergency Platform
From IDC to Cloud
New Platform Design
15. New Platform Design
• Unified API
• Personalization
• Scalable Performance
Emergency Platform
From IDC to Cloud
New Platform Design
Unified API
Personalization
Scalable
16. New Platform Design
• Unified API model
Eliminate duplicated resource allocation
Same behavior on all client
• Requirements
Restful API
JSON output
Specification & Documentation
Testing
Auto server generation from spec
ASP.Net
MAPI
Unified API
Emergency Platform
From IDC to Cloud
New Platform Design
Unified API
Personalization
Scalable
17. Unified API
• Swagger (http://swagger.io/)
Restful API definition with .yaml
Free documentation, testing tools available
Auto-generate client & server for several languages
Emergency Platform
From IDC to Cloud
New Platform Design
Unified API
Personalization
Scalable
18. Unified API - Swagger
Emergency Platform
From IDC to Cloud
New Platform Design
Unified API
Personalization
Scalable
19. Unified API-Swagger
API Design
(.yaml)
Validate &
Test
(Swagger
editor)
Commit
(To GitHub)
Server Code
Gen
(Custom Tool)
API Doc
Site
Publishing
Emergency Platform
From IDC to Cloud
New Platform Design
Unified API
Personalization
Scalable
21. Unified API-Auto Code Gen
Emergency Platform
From IDC to Cloud
New Platform Design
Unified API
Personalization
Scalable
22. New Platform Design
• Personalization
Continues viewing – Across N-Screen
Popular content listing
Your program listing
All based on “Bookmark” concept
Client send streaming status to server
Every 10 second
UDP and HTTP based protocol support
Heavy load on server side
Emergency Platform
From IDC to Cloud
New Platform Design
Unified API
Personalization
Scalable
25. Scalable Design
• All participant server must be able to increase performance by
increasing number of VMs
Rules out MSSQL as online database
Active/Active is not enough
MongoDB is choice of database (as mostly read-cache)
Cassandra NoSQL as data logging server
• Single VM failure should not affect whole platform’s functionality
Utilize lots of Load Balancer
Emergency Platform
From IDC to Cloud
New Platform Design
Unified API
Personalization
Scalable
26. Selection of NoSQL
Emergency Platform
From IDC to Cloud
New Platform Design
Unified API
Personalization
Scalable
28. Scalable Design
Client
Cassandra
API Server
Apache SparkMongo DB
Service
MS SQL
Pentaho
Kettle
Batch Update
Emergency Platform
From IDC to Cloud
New Platform Design
Unified API
Personalization
Scalable
29. MongoDB Scale Out
Mongo DB
(Master,
WRITE)
Mongo DB
(Slave, READ)
Mongo DB
(Slave, READ)
Mongo DB
(Slave, READ)
Mongo DB
(Slave, READ)
Mongo DB
(Slave, READ)
Up To 50 replica members
Emergency Platform
From IDC to Cloud
New Platform Design
Unified API
Personalization
Scalable
30. MongoDB
(Slave)
Scalable Design
Emergency Platform
From IDC to Cloud
New Platform Design
Unified API
Personalization
Scalable
Web Servers
(Static Page)
LB Group
Web Servers
(Static Page)
Web Servers
(Static Page)
Web Servers
(Static Page)
Web Servers
(Static Page)
LB Group
Web Servers
(Static Page)
Web Servers
(Static Page)API Servers
MongoDB
(Slave)
MongoDB
(Master)
Cassandra
Ring
Node
Node
Node
Node
Node
Node
Web Servers
(Static Page)
LB Group
Web Servers
(Static Page)
Web Servers
(Static Page)
Bookmark
Collectors
31. The Result?
• POOQ obtain exclusive Internet streaming rights on Premier12 Baseball Game
• 19th November, 2015 - Premier12, Korea vs. Japan
32. The Result
• 551G Network
• 283,577 Users
• Platform was stable
CPU Utilization was 20%
33. The Road So Far…
• Built solid platform foundation
• With collaboration of fantastic developer team
• Confidence on handling large user interaction
• Still far way to go, though
34. Lesson Learned (About Cloud)
• Very efficient, Will recommend to almost any platform
• Enjoying decent support in general
• VMs are really slower than you think
• Not all VMs are equal
• Sometimes your bottleneck is not number of VMs
• Long way from perfection/shit happen