Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

LINE LIVE のチャットが
30,000+/min のコメント投稿を捌くようになるまで

3,621 views

Published on

JJUG CCC 2018 Spring

Published in: Technology
  • Be the first to comment

LINE LIVE のチャットが
30,000+/min のコメント投稿を捌くようになるまで

  1. 1. LINE LIVE 
 30,000+/min Go Hagiwara (Oklahomer) , Dev3 Center
  2. 2. ● Oklahomer
 or ● @ LINE ● LINE LIVE API ● ● ● Bot Framework ChatOps bot ● Java/Golang/Python/Perl
  3. 3. ● LINE ● ● ● ● LINE LIVE
  4. 4. ● MC ● ● LIVE
  5. 5. ● ● ● ●
  6. 6. WebSocket - Server-Client akka actor model - Server Redis Cluster - Pub/Sub Server 
 KVS
  7. 7. subtitle TITLE
  8. 8. WebSocket
  9. 9. akka Actor Model
  10. 10. Pub/sub Server Lua scripting atomic Redis Cluster
  11. 11. WebSocket - Server-Client akka actor model - Server Redis Cluster - Server
  12. 12. akka Actor Model
  13. 13. ● Perl ● Java ● Effective Java ● Java ● Java ●
  14. 14. ● Parallel::Prefork ● Parallel::ForkManager ● java.util.concurrent package ● ParallelStream Perl v.s. Java
  15. 15. akka Actor Model ● actor ● actor ● ● Actor ● ● ● actor ●
  16. 16. Actor
  17. 17. Actor ● Actor mailbox ● mailbox ● mailbox ● ● Actor ● actor mailbox ● ●
  18. 18. ● Actor ● ●
  19. 19. Supervisor
  20. 20. Supervisor ● actor actor ● ● actor actor ● let-it-crash actor “crash” ●
  21. 21. Supervisor Strategy ● actor ● ● crash actor ●
  22. 22. Supervisor Strategy - ● One-for-One strategy ● crash actor ● All-for-One strategy ● actor ● actor actor actor
  23. 23. Supervisor Strategy - Directives ● Restart - actor mailbox ● Crash actor actor restart ● ● preRestart hook → ● Resume - mailbox 
 ●
  24. 24. Supervisor Strategy - Directives ● Stop - actor ● ● actor ● Escalate - Supervisor Actor ● Supervisor Strategy 
 Child Actor → Parent Actor → Grandparent Actor
  25. 25. Supervisor Strategy - Directives
  26. 26. Supervisor Strategy - 1 public class MyActor extends UntypedActor { 2 private static SupervisorStrategy strategy = 3 new OneForOneStrategy(10, Duration.create("1 minute"), t -> { 4 if (t instanceof ActorInitializationException) { 5 return stop(); 6 } else if (t instanceof ActorKilledException) { 7 return stop(); 8 } else if (t instanceof Exception) { 9 return restart(); 10 } 11 12 return escalate(); 13 }); 14 15 @Override 16 public SupervisorStrategy supervisorStrategy() { 17 return strategy; 18 } 19 20 @Override 21 public void onReceive(Object o) throws Exception { 22 } 23 }
  27. 27. Actor
  28. 28. Actor ● Actor ActorRef ● ● ● ● etc… ● actor dead letter ●
  29. 29. Actor ● ● watch actor ● Terminated ● unwatch ● ● Actor crash actor
  30. 30. Chat Server Actor
  31. 31. ChatSupervisor ● 1 JVM 1 ● actor ● ” ” actor ●
  32. 32. ChatRoomActor ● ● ● Redis pub/sub ● UserActor ● ● ● UserActor ● ●
  33. 33. UserActor ● ● WebSocket ● ChatRoomActor ●
  34. 34. ● ChatRoomActor UserActor ChatSupervisor ● ” ” UserActor ChatRoomActor ● WebSocket UserActor ● ChatRoomActor crash → restart UserActor restart ● ChatRoomActor UserActor
  35. 35. ● ChatRoomActor UserActor ● ChatRoomActor UserActor watch ● UserActor WebSocket UserActor stop ● ChatRoomActor Terminated ● ChatRoomMembers ● ● Watch Terminated ● ChatRoomActor restart ● ChatRoomActor crash UserActor
  36. 36. ● ● Restart 1 public static Props props(final AggregatedChannel channel, 2 final ChatRoom chatRoom, 3 final ChatRoomMembers chatRoomMembers) { 4 // Returns a property to construct ChatRoomActor 5 // NOTE: Those objects passed to actor's constructor are inherited on restart 6 // such as ChatRoomMembers that caches belonging UserActors 7 return Props.create(ChatRoomActor.class, 8 () -> new ChatRoomActor(channel, 9 chatRoom, 10 chatRoomMembers)); 11 }
  37. 37. ● actor ● actor ● actor Actor System ● ● Actor model
  38. 38. ● Controller ● validate ● ChatSupervisor ChatRoomActor ● WebSocket ● WebSocket UserActor ● ChatRoomActor WebSocket
  39. 39. ● Controller / UserActor WebSocket ● ● UserActor crash ● Supervisor Strategy restart 
 resume ● Validation ● ● UserActor
  40. 40. ● ● Redis ● WebSocket ● ● actor ●
  41. 41. - Blocking API ● WebSocket Jetty issue ● https://github.com/eclipse/jetty.project/issues/272 ● UserActor ● ● ● JVM ● Spring API ● Jetty issue
  42. 42. - Blocking API ● ThreadPool ● Jetty issue ● Watchdog ● ● ● UserActor ● ChatRoomActor Terminated 
 Actor
  43. 43. ● ● ● ● ● ● ● ●
  44. 44. ● ● Redis Cluster pub/sub ● key ChatRoomActor pub/sub ● ChatRoomActor UserActor
  45. 45. ● Remote Actor ● WebSocket UserActor ● ChatRoomActor ChatSupervisor, UserActor ● Deploy Redis pub/sub
  46. 46. Redis ● Actor Redis Pub/ Sub ● Pub/Sub ● Subscribe ● ● Actor
  47. 47. Lettuce ● Redis Cluster ● Master/slave failover MOVED/ASK node ● API ● Pub/sub subscribe failover ● Lua scripting
  48. 48. Lettuce ● ● Redis Cluster reshard 
 validation ● Redis Cluster → application ● Failover reshard ● Periodic refresh ● Adaptive refresh
  49. 49. Lettuce 1 /* 2 To get rid of stale connection and establish new connection when crush/failover occurs, 3 refresh its topologyView periodically. 4 */ 5 ClusterTopologyRefreshOptions topologyRefreshOptions = 6 ClusterTopologyRefreshOptions.builder() 7 .enablePeriodicRefresh(30, TimeUnit.SECONDS) 8 .enableAllAdaptiveRefreshTriggers() 9 .build(); 10 /* 11 Basically we use the default settings, but to avoid connection error caused by newly added server, 12 override the validateClusterNodeMembership and skip validation for new server. 13 */ 14 ClusterClientOptions clusterClientOptions = 15 ClusterClientOptions.builder() 16 .validateClusterNodeMembership(false) 17 .topologyRefreshOptions(topologyRefreshOptions) 18 .build();
  50. 50. ● WebSocket ● Server - Client ● akka Actor Model ● ● Actor Actor ● Blocking API ● Redis Cluster ● ●
  51. 51. ● LINE LIVE ● ● https://engineering.linecorp.com/ja/blog/detail/85 ● Yet another Akka introduction for dummies ● ● http://blog.oklahome.net/2016/01/akka-introduction-for-dummies.html ● How to cook lettuce @Java casual ● Lettuce ● https://www.slideshare.net/Oklahomer/how-to-cook-lettuce-java- casual
  52. 52. THANK YOU

×