Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How does LINE effectively handle media content?

414 views

Published on

Hae Seong Choi
LINE+ / Media Platform

LINE has dozens of services as well as LINE messenger. New services and projects are created every year. Media Platforms help these numerous projects store and reliably deliver media content to users. Through the media platform, the service development team can focus on developing their own service functions.
Media platforms handle a huge amount of traffic every day and store data. My team has had a lot of concerns about how to store efficiently, how to transmit quickly and reliably, and I will introduce some of them.
And we have technology not only for storage and transmission, but also for media processing. Among them, I would like to introduce you to the live broadcasting platform, which is the basic technology of LINE LIVE.
And I will introduce the Vision AI technology provided by the media platform.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

How does LINE effectively handle media content?

  1. 1. HOW DOES LINE EFFECTIVELY HANDLE MEDIA CONTENT?
  2. 2. Topics • Media Platform of LINE • Experiences and Issues • Goals for 2019
  3. 3. LINE + MEDIA PLATFORM
  4. 4. LINE JAPAN
  5. 5. Users TrafficStorage Services LINE RAPID GROWTH ETC Timeline Keep LINE
 Play Album
  6. 6. • Dealing with 
 the increasing amount of storage and traffic flexibly • Service settings can be added and changed easily • Stores user content effectively and reliably PLATFORM REQUIREMENTS
  7. 7. • Content accessible 
 only through a service’s authentication API • Minimal dependency on services • No knowledge of user and content information IMPORTANT VALUES AS A PLATFORM
  8. 8. GLOBAL POP JAPAN TAIWAN SINGAPORE THAILAND KOREA US GERMANY
  9. 9. SUPPORT ALL LINE SERVICES TAIWAN SINGAPORE THAILAND KOREA GERMANY US JAPAN
  10. 10. TRAFFIC TAIWAN SINGAPORE THAILAND GERMANY 500Gbps KOREA US JAPAN
  11. 11. TRAFFIC + STORAGE TAIWAN SINGAPORE THAILAND GERMANY 500Gbps KOREA US JAPAN 70PB
  12. 12. TRAFFIC + STORAGE TAIWAN SINGAPORE THAILAND GERMANY 500Gbps KOREA US JAPAN 200PB
  13. 13. TECH EXTENSION
  14. 14. TECH EXTENSION Feature Name: Value Adult Score : 0.991797 Adult : NORMAL ADULT:0.018685 NORMAL:0.970296 SEXY:0.011020
  15. 15. Experiences and Issues Storage Live Broadcasting 
 Delivery 
 Vision AI 
 
 MEDIA PLATFORM Quick
 Stable Effective Reliable In-house Live Broadcasting Platform Media content
 analysis
  16. 16. Experiences and Issues Storage Live Broadcasting 
 Delivery 
 Vision AI 
 
 MEDIA PLATFORM Quick
 Stable Effective Reliable In-house Live Broadcasting Platform Media content
 analysis
  17. 17. BEGINNING OF LINE
  18. 18. LINE BECAME GLOBALIZED Too far !!! Too far !!!
  19. 19. THAILAND IN 2013 Too far !!! Too far !!! 2013 TH(3g) 100k Image send 3~4 seconds Sometimes more than 10 seconds
  20. 20. SOLUTION Too far !!! Too far !!! Reduce the latency
  21. 21. Too much traffic !!! TOO MUCH TRAFFIC Too much traffic !!! Storage overload
  22. 22. Too much traffic !!! SOLUTION Too much traffic !!! Must protect
 main storage
  23. 23. ONE SOLUTION Caching
  24. 24. FRONTEND AND CACHE IN GLOBAL POP JAPAN TAIWAN SINGAPORE THAILAND KOREA GERMANY Internet line Leased line US
  25. 25. PROCESS USER TRAFFIC FROM FRONTEND JAPAN TAIWAN SINGAPORE THAILAND KOREA GERMANY Frontend + Cache Leased line Backend 
 Main storage Internet
 line Amount
 of 
 traffic Distance US
  26. 26. CACHE IN ISP Frontend + Cache Backend + Main storage LISA LISA ISP : Internet Service Provider LISA : LIne Site Accelerator LISA LISA
  27. 27. LISA Frontend + Cache Backend + Main storage LISA LISA LISA LISA
  28. 28. LISA Frontend + Cache Backend + Main storage ISP ISP ISP 80Gbps 60Gbps
  29. 29. LISA FOR ISP Frontend + Cache Backend + Main storage 80Gbps 60Gbps Less than 
 50Mbps Less than 
 40Mbps LISA LISA LISA WITH LISA • ISP • Can reduce cost • Can control traffic
  30. 30. LISA FOR LINE Frontend + Cache Backend + Main storage 80Gbps 60Gbps Less than 
 50Mbps Less than 
 40Mbps LISA LISA LISA WITH LISA • ISP + LINE • Can reduce cost • Can control traffic
  31. 31. LISA FOR USER Frontend + Cache Backend + Main storage 80Gbps 60Gbps Less than 
 50Mbps Less than 
 40Mbps LISA LISA LISA WITH LISA • ISP + LINE • Can reduce cost • Can control traffic • User • Faster response time
  32. 32. What’s special about our platform? CACHE POLICY
  33. 33. 1 User A User B Main storage CACHE 1:1 CHAT ROOM In a 1:1chat room, it is better not to cache 2
  34. 34. User A User B Main storage Slow!!! 1:1 CHAT ROOM Should we not use the cache at all in 1:1 chat rooms? CACHE NO CACHE 1 5 3 2 4
  35. 35. Origin User A User B CACHE WHEN CONTENT IS UPLOADED The uploaded content is stored in both main storage and cache CACHE 2 2 1
  36. 36. Origin User A User B 4 ALWAYS DOWNLOAD FROM CACHE Always HIT CACHE 2 2 CACHE HIT 1 3 Quick!!!
  37. 37. HOT OBJECTS ?? ?? ? ?
  38. 38. HOT OBJECTS Too many downloads Storage 
 overload
  39. 39. THIS KIND OF SITUATION CAN HAPPEN Too many downloads AGAIN AT ANY TIME
  40. 40. AUTOMATIC Detection • Scope : Certain resources • Time : Within L seconds • Condition : Downloaded
 N or more times HOT OBJECT DETECTION HOT OBJECT
  41. 41. AUTOMATICALLY CHANGE TO MOST EFFECTIVE POLICY Change Cache Policy • Cache : Inactive >> Active • Hashing >> Round Robin Protect 
 Cache & Storage CACHE
  42. 42. Experiences and Issues Storage Live Broadcasting 
 Delivery 
 Vision AI 
 
 MEDIA PLATFORM Quick
 Stable Effective Reliable In-house Live Broadcasting Platform Media content
 analysis
  43. 43. ServerUser RELIABLE STORAGE
  44. 44. UserCopy #2 Copy #1 ServerUser Copy #3 THREE-COPY POLICY
  45. 45. 3 COPIES COST ₩ ₩₩ MORE STORAGE COST
  46. 46. 3 COPIES COST ₩ ₩₩ REDUCE STORAGE TO REDUCE COST
  47. 47. Accesscount Period Storageusage STATISTICS
  48. 48. Accesscount Period Storageusage Pareto principle STATISTICS
  49. 49. Accesscount Period Storageusage Long tail principle STATISTICS
  50. 50. High
 performance
 storage Standard 
 storage High-density storage 3 TYPES OF STORAGE ServerUser
  51. 51. Access count High
 performance
 storage Standard 
 storage High-density storage 3 TYPES OF STORAGE ServerUser
  52. 52. CAPACITY PERFORMANCE PRICE SAVE STORAGE COST High
 performance
 storage Standard 
 storage High-density storage
  53. 53. IDC server rack IDC server rack SAVE IDC SPACE
  54. 54. Let’s do something !!! IDC server rack ANTMAN
  55. 55. ROLE OF ANTMAN
  56. 56. MP4 meta (Including HEIF custom header) Image meta
 (Exif, XMP) HEVC
 (a.k.a. H265) HEIF(MP4) JPEG (since 1992) HEVC (since 2013) 50% CAPACITY COMPARED TO JPEG
  57. 57. JPEG headers Image meta
 (Exif, XMP) Scan data JPEG MP4 meta (Including HEIF custom header) Image meta
 (Exif, XMP) HEVC
 (a.k.a. H265) FIND SUITABLE COMPRESSION RATE AUTOMATICALLY HEIF(MP4)
  58. 58. HEVC
 (a.k.a. H265) Q factor Width RECUDE NUMBER OF HEVC ENCODING ATTEMPTS MP4 meta (Including HEIF custom header) Image meta
 (Exif, XMP) HEIF(MP4) … Height
  59. 59. ? NOT ALL DEVICES CAN DISPLAY HEIF
  60. 60. REVERT HEIF TO JPEG IN REAL TIME
  61. 61. BACKUP ORIGINAL QUALITY FACTOR JPEG headers Image meta
 (Exif, XMP) Scan data JPEG MP4 meta (Including HEIF custom header) Image meta
 (Exif, XMP) HEVC
 (a.k.a. H265) HEIF(MP4) Quality factor Quality factor
  62. 62. Experiences and Issues Storage Live Broadcasting 
 Delivery 
 Vision AI 
 
 MEDIA PLATFORM Quick
 Stable Effective Reliable In-house Live Broadcasting Platform Media content
 analysis
  63. 63. LIVE PLATFORM - BEGINNING In-house development Open
 Source Commercial software
  64. 64. Open source Commercial software In-house development + + LIVE PLATFORM - BEGINNING
  65. 65. ISSUES USING EXTERNAL SOLUTION Open source Commercial software In-house development + + Issues
  66. 66. ISSUES USING EXTERNAL SOLUTION “ We have experienced failures in live stream publishing. Is there a problem with the server? Please check. ”
  67. 67. “ We have experienced failures in live stream publishing. Is there a problem with the server? Please check. ” “ We are experiencing video and audio synchronization problem. What is the problem? ” ISSUES USING EXTERNAL SOLUTION
  68. 68. “ We have experienced failures in live stream publishing. Is there a problem with the server? Please check. ” “ We are experiencing video and audio synchronization problem. What is the problem? ” “ It takes too long to load. Can you please improve it? ” ISSUES USING EXTERNAL SOLUTION
  69. 69. “ We have experienced failures in live stream publishing. Is there a problem with the server? Please check. ” “ We are experiencing video and audio synchronization problem. What is the problem? ” “ It takes too long to load. Can you please improve it? ” “ The service infra costs too much. Can you reduce the cost by improving performance? ” ISSUES USING EXTERNAL SOLUTION
  70. 70. “ We have experienced failures in live stream publishing. Is there a problem with the server? Please check. ” “ We are experiencing video and audio synchronization problem. What is the problem? ” “ It takes too long to load. Can you please improve it? ” “ The service infra costs too much. Can you reduce the cost by improving performance? ” ISSUES USING EXTERNAL SOLUTION
  71. 71. LINE MEDIA DELIVERY SERVER WITH RTMP LIME
  72. 72. LIME - TWO IMPROVEMENTS Improve RTMP flow with RTMP SDK
  73. 73. RTMP FLOW TCP handshake
  74. 74. RTMP FLOW TCP handshake TLS Handshake
  75. 75. RTMP FLOW TCP handshake RTMP handshake TLS Handshake
  76. 76. RTMP FLOW TCP handshake RTMP handshake RTMP command TLS Handshake
  77. 77. RTMP FLOW TCP handshake RTMP handshake RTMP command RTMP “play” command TLS Handshake
  78. 78. RTMP FLOW TCP handshake RTMP handshake RTMP command RTMP “play” command Audio video data TLS Handshake
  79. 79. RTMP FLOW TCP handshake RTMP handshake RTMP command RTMP “play” command Audio video data TLS Handshake At least 7 times
  80. 80. DELAY 250ms RTMP 
 edge server
  81. 81. DELAY 250ms * 7 = 1.75 sec RTMP 
 edge server
  82. 82. LIME - IMPROVED RTMP FLOW TCP handshake TLS handshake RTMP handshake RTMP command DONE DONE DONE DONE
  83. 83. LIME - IMPROVED RTMP FLOW TCP handshake TLS handshake RTMP handshake RTMP command “Play” command Just once
  84. 84. LIME - TWO IMPROVEMENTS Improve RTMP flow with RTMP SDK Cache previous frame’s data set
  85. 85. LIME - PREVIOUS FRAME CACHE I-frame P-frame P-frame P-frame P-frame P-frame P-frame P-frame P-frame I-frame P-frame + + + + + + + + +X
  86. 86. LIME - PREVIOUS FRAME CACHE I-frame P-frame P-frame P-frame P-frame P-frame P-frame P-frame P-frame I-frame P-frame +
  87. 87. LIME - PREVIOUS FRAME CACHE + I-frame P-frame P-frame P-frame P-frame P-frame P-frame P-frame P-frame I-frame P-frame + + + + + + + +
  88. 88. LIME - ADVANCED TECH Improve RTMP flow with RTMP SDK Cache previous frame’s data set Less delay
  89. 89. LIVE PLATFORM JP
  90. 90. LIVE PLATFORM with Quiz
  91. 91. Experiences and Issues Storage Live Broadcasting 
 Delivery 
 Vision AI 
 
 MEDIA PLATFORM Quick
 Stable Effective Reliable In-house Live Broadcasting Platform Media content
 analysis
  92. 92. 3.5 15 Operators Contents X X
  93. 93. ! Adult content filter ! Symbol detector ! QR / BAR code detector ! AD image filter
  94. 94. OCR Japanese English Chinese Korean
 Thai Indonesian+ +
  95. 95. 2019 MEDIA PLATFORM
  96. 96. 2019 webRTC Open source LIME
  97. 97. 2019 Enhanced media processing using GPU webRTC Open source LIME New image- processing server
  98. 98. 2019 Enhanced media processing using GPU New image- processing server Operation toolswebRTC PaaS Open source LIME
  99. 99. THANK YOU

×