Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Trial and Error in 
Releasing GREE Chat 
Shun Ozaki, Takayuki Hasegawa 
Copyright © GREE, Inc. All Rights Reserved. 
S...
Copyright © GREE, Inc. All Rights Reserved. 
Self Introduction 
• Shun Ozaki 
• @wozaki 
• Joined in April, 2013 
• Androi...
Copyright © GREE, Inc. All Rights Reserved. 
GREE Chat 
• Released at June 2 
2/56
How to build our system for hundreds of 
thousands daily users 
Copyright © GREE, Inc. All Rights Reserved. 
Agenda 
3/56
Goal 
Share our knowledge we got through 
the development of GREE Chat 
Copyright © GREE, Inc. All Rights Reserved. 
• Why...
Copyright © GREE, Inc. All Rights Reserved. 
Outline 
• Reason of Selecting Scala 
• Learning Scala in a Team 
• Architect...
• Reason of Selecting Scala 
• Learning Scala in a Team 
• Architecture of GREE Chat 
• Obstacles 
• Summary 
Copyright © ...
Decide the Language 
based on the Requirement 
Copyright © GREE, Inc. All Rights Reserved.
Requirements 
• Hundreds of thousands daily users 
• Real-time response 
Copyright © GREE, Inc. All Rights Reserved. 
 Co...
Copyright © GREE, Inc. All Rights Reserved. 
◎ GREE uses PHP so heavily 
• Many libraries and know-how in GREE 
△ Streamin...
◎Compatible with concurrent programming 
◎One process, many connections 
◎High maintainability 
• Static typing 
• Functio...
• Reason of Selecting Scala 
• Learning Scala in a Team 
• Architecture of GREE Chat 
• Obstacles 
• Summary 
Copyright © ...
After adopting Scala … 
Copyright © GREE, Inc. All Rights Reserved.
Only two Scala programmers in a team of seven 
Copyright © GREE, Inc. All Rights Reserved. 
Shortage of Scala experts 
2/ ...
Only two Scala programmers in a team of seven 
We must build up our skill! 
Copyright © GREE, Inc. All Rights Reserved. 
S...
Copyright © GREE, Inc. All Rights Reserved. 
Learning of Scala 
1. Self study 
2. Study club 
3. Pair programming 
15/56
Copyright © GREE, Inc. All Rights Reserved. 
1. Self Study 
Understand the basic syntax
Copyright © GREE, Inc. All Rights Reserved. 
1. Self Study 
Books 
17/56
Copyright © GREE, Inc. All Rights Reserved. 
1. Self Study 
Documents by Twitter 
https://twitter.github.io/scala_school/ ...
Copyright © GREE, Inc. All Rights Reserved. 
1. Self Study 
Source code from OSS 
19/56
Copyright © GREE, Inc. All Rights Reserved. 
2. Study Club 
Share what we learned by ourselves 
Introduction to functional...
2. Study Club 
• Each engineer solves the problems 
provided by Scala experts 
Copyright © GREE, Inc. All Rights Reserved....
• Example of factorial generation algorithm 
Copyright © GREE, Inc. All Rights Reserved. 
2. Study Club 
Discussion with M...
• Example of factorial generation algorithm 
Use var Use recursion 
Copyright © GREE, Inc. All Rights Reserved. 
2. Study ...
3. Pair Programming 
Practice by using the knowledge 
Copyright © GREE, Inc. All Rights Reserved. 
obtained from study clu...
Copyright © GREE, Inc. All Rights Reserved. 
3. Pair Programming 
• Effective learning 
• Learn about symbols (e.g., +, @)...
Copyright © GREE, Inc. All Rights Reserved. 
Summary: Adopting Scala 
• Reasons of selecting Scala 
• Compatible with conc...
• Reason of Selecting Scala 
• Learning Scala in a Team 
• Architecture of GREE Chat 
• Obstacles 
• Summary 
Copyright © ...
Architecture of Backend (simplified) 
Copyright © GREE, Inc. All Rights Reserved. 
• Servers separated by Queues 
• API Se...
Role of Each Server 
Copyright © GREE, Inc. All Rights Reserved. 
and Framework
Copyright © GREE, Inc. All Rights Reserved. 
API Server 
Process users’ requests 
• Enqueue events 
• e.g., Send message 
...
• RPC system for JVM based on Netty 
• Support us to write asynchronous logic 
Copyright © GREE, Inc. All Rights Reserved....
Copyright © GREE, Inc. All Rights Reserved. 
EventBus Server 
Process events in Queue 
• Examples of tasks 
• Logging 
• I...
Framework to write concurrent, distributed 
logic more easily 
• No need to handle shared resources 
Copyright © GREE, Inc...
Copyright © GREE, Inc. All Rights Reserved. 
Example of Akka 
34/56
Copyright © GREE, Inc. All Rights Reserved. 
Example of Akka 
Match by Event type 
35/56
Copyright © GREE, Inc. All Rights Reserved. 
Example of Akka 
Send copy to child 
36/56
Copyright © GREE, Inc. All Rights Reserved. 
Example of Akka 
log 
37/56
Copyright © GREE, Inc. All Rights Reserved. 
Stream Server 
Supply events to connected users 
• Connect by streaming 
• In...
Copyright © GREE, Inc. All Rights Reserved. 
Summary: Architecture 
• Each server is separated by Queue 
• API, EventBus, ...
Copyright © GREE, Inc. All Rights Reserved. 
Outline 
• Reason of Selecting Scala 
• Learning Scala in a Team 
• Architect...
Copyright © GREE, Inc. All Rights Reserved. 
Obstacles 
1.JVM 
• Full GC 
2. Self-created Scala library 
• Sharding 
• ID ...
Full GC 
Problems 
Copyright © GREE, Inc. All Rights Reserved.
• Scala runs on JVM 
• We want to prevent “stop the world” 
• GC in young generation is faster than full GC 
• Try not to ...
Copyright © GREE, Inc. All Rights Reserved. 
Causes of Uncollected 
Reference Problem 
• Use method that has side effects ...
Copyright © GREE, Inc. All Rights Reserved. 
Causes of Uncollected 
Reference Problem 
• Use method that has side effects ...
Copyright © GREE, Inc. All Rights Reserved. 
Short-Lived Object 
• Use val variable (final variable) 
• Don’t use mutable ...
Sharding: Scalable Storage 
47/56 
Copyright © GREE, Inc. All Rights Reserved.
Copyright © GREE, Inc. All Rights Reserved. 
Scale up? 
48/56
Copyright © GREE, Inc. All Rights Reserved. 
Scale out! 
49/56
Copyright © GREE, Inc. All Rights Reserved. 
Horizontal Partitioning 
• Sharding is necessary! 
• Considering capacity and...
ID Architecture: 
UUID and InnoDB 
Ref:MySQL InnoDB Primary Key Choice: GUID/UUID vs Integer Insert Performance 
http://kc...
Ref:MySQL InnoDB Primary Key Choice: GUID/UUID vs Integer Insert Performance 
http://kccoder.com/mysql/uuid-vs-int-insert-...
Drop UUID, Take Time-Based 
• At first, we used UUID as primary key 
• But this caused MySQL to insert randomly 
• Created...
Copyright © GREE, Inc. All Rights Reserved. 
Summary: Obstacles & 
Workaround 
• JVM 
• Use short-lived objects to avoid F...
Copyright © GREE, Inc. All Rights Reserved. 
Outline 
• Reason of Selecting Scala 
• Learning Scala in a Team 
• Architect...
Copyright © GREE, Inc. All Rights Reserved. 
Summary 
• Reasons for selecting Scala 
• Compatible with concurrent programm...
Copyright © GREE, Inc. All Rights Reserved. 
Thanks! 
@tomoyoshi_ogura, @kyo_ago, @takc923 
@yoshie_777, @le_chang, @beket...
Upcoming SlideShare
Loading in …5
×

[ScalaMatsuri] グリー初のscalaプロダクト!チャットサービス公開までの苦労と工夫

4,439 views

Published on

ScalaMatsuriでの講演資料です。
http://scalamatsuri.org/

<セッション内容>
グリーでは今年6月にGREEチャットというサービスをリリースしました。
本発表では、GREEチャットのバックエンドをScalaで開発したときの利点や苦労した点についてお話させて頂きます。
トピックは以下の通りです。

-弊社で初めてScalaのプロダクトを導入する際の苦労
-数十万DAUでもリアルタイム性を維持するための工夫
-AkkaやFinagleなどのOSSライブラリを使った非同期処理、並列処理

グリー株式会社 長谷川 貴之&尾崎 俊

Published in: Technology
  • Be the first to comment

[ScalaMatsuri] グリー初のscalaプロダクト!チャットサービス公開までの苦労と工夫

  1. 1. The Trial and Error in Releasing GREE Chat Shun Ozaki, Takayuki Hasegawa Copyright © GREE, Inc. All Rights Reserved. Scala Matsuri2014 B-6 GREE's First Scala Product
  2. 2. Copyright © GREE, Inc. All Rights Reserved. Self Introduction • Shun Ozaki • @wozaki • Joined in April, 2013 • Android Application • Takayuki Hasegawa • @hase1031 • Joined in April, 2013 • NLP, Machine Learning 1/56
  3. 3. Copyright © GREE, Inc. All Rights Reserved. GREE Chat • Released at June 2 2/56
  4. 4. How to build our system for hundreds of thousands daily users Copyright © GREE, Inc. All Rights Reserved. Agenda 3/56
  5. 5. Goal Share our knowledge we got through the development of GREE Chat Copyright © GREE, Inc. All Rights Reserved. • Why Scala? • Team development • Decision of frameworks • Obstacles & workarounds 4/56
  6. 6. Copyright © GREE, Inc. All Rights Reserved. Outline • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary
  7. 7. • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Copyright © GREE, Inc. All Rights Reserved. Outline
  8. 8. Decide the Language based on the Requirement Copyright © GREE, Inc. All Rights Reserved.
  9. 9. Requirements • Hundreds of thousands daily users • Real-time response Copyright © GREE, Inc. All Rights Reserved.  Connect with streaming • Run on a small number of servers  Utilize server resource effectively • Maintain for over 5 years 8/56
  10. 10. Copyright © GREE, Inc. All Rights Reserved. ◎ GREE uses PHP so heavily • Many libraries and know-how in GREE △ Streaming • #connections = #processes △ Single thread, multi process • Overhead of spawning an OS process △ Maintainability • Dynamic typing 9/56
  11. 11. ◎Compatible with concurrent programming ◎One process, many connections ◎High maintainability • Static typing • Functional programming (no side effects) ◎Stimulate new technology learning in GREE • Open to new languages & technologies! Copyright © GREE, Inc. All Rights Reserved. Scala! 10/56
  12. 12. • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Copyright © GREE, Inc. All Rights Reserved. Outline
  13. 13. After adopting Scala … Copyright © GREE, Inc. All Rights Reserved.
  14. 14. Only two Scala programmers in a team of seven Copyright © GREE, Inc. All Rights Reserved. Shortage of Scala experts 2/ 7 13/56
  15. 15. Only two Scala programmers in a team of seven We must build up our skill! Copyright © GREE, Inc. All Rights Reserved. Shortage of Scala experts 2/ 7 14/56
  16. 16. Copyright © GREE, Inc. All Rights Reserved. Learning of Scala 1. Self study 2. Study club 3. Pair programming 15/56
  17. 17. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study Understand the basic syntax
  18. 18. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study Books 17/56
  19. 19. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study Documents by Twitter https://twitter.github.io/scala_school/ http://twitter.github.io/effectivescala/ 18/56
  20. 20. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study Source code from OSS 19/56
  21. 21. Copyright © GREE, Inc. All Rights Reserved. 2. Study Club Share what we learned by ourselves Introduction to functional programming
  22. 22. 2. Study Club • Each engineer solves the problems provided by Scala experts Copyright © GREE, Inc. All Rights Reserved. • e.g., Binary Tree, Fibonacci Sequence 21/56
  23. 23. • Example of factorial generation algorithm Copyright © GREE, Inc. All Rights Reserved. 2. Study Club Discussion with Members Use var Use recursion 22/56
  24. 24. • Example of factorial generation algorithm Use var Use recursion Copyright © GREE, Inc. All Rights Reserved. 2. Study Club Discussion with Members No side effect 23/56
  25. 25. 3. Pair Programming Practice by using the knowledge Copyright © GREE, Inc. All Rights Reserved. obtained from study club
  26. 26. Copyright © GREE, Inc. All Rights Reserved. 3. Pair Programming • Effective learning • Learn about symbols (e.g., +, @), which cannot be searched easily on the Internet • Supplement the knowledge not taught at study club • Complex syntaxes (e.g., implicit, currying) • Development tools (e.g., sbt, IntelliJ) 25/56
  27. 27. Copyright © GREE, Inc. All Rights Reserved. Summary: Adopting Scala • Reasons of selecting Scala • Compatible with concurrent programming • Maintainability because of static typing • Learning of Scala • Self Study, Study Club, Pair Programming • Big burdens on Scala experts • Needed for teaching team members • Maintain the quality of codes 26/56
  28. 28. • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Copyright © GREE, Inc. All Rights Reserved. Outline
  29. 29. Architecture of Backend (simplified) Copyright © GREE, Inc. All Rights Reserved. • Servers separated by Queues • API Server: Process users’ requests • EventBus Server: Process events in Queue • Stream Server: Supply event to connected users 28/56
  30. 30. Role of Each Server Copyright © GREE, Inc. All Rights Reserved. and Framework
  31. 31. Copyright © GREE, Inc. All Rights Reserved. API Server Process users’ requests • Enqueue events • e.g., Send message Join/leave conversation • Techniques to process a lot of requests • Delegate heavy tasks (e.g., Disk I/O) to others • Logic is written to run asynchronously • scala.concurrent.Future 30/56
  32. 32. • RPC system for JVM based on Netty • Support us to write asynchronous logic Copyright © GREE, Inc. All Rights Reserved. • Used by large scale web services • e.g., Twitter, Tumblr, Foursquare, Pinterest • Equipped with various clients • Redis, Memcached • With retry policy, connection pools 31/56
  33. 33. Copyright © GREE, Inc. All Rights Reserved. EventBus Server Process events in Queue • Examples of tasks • Logging • Inquire GREE internal system • Store events in DB • etc. • Problems of concurrent processing • Deadlock, race condition, … 32/56
  34. 34. Framework to write concurrent, distributed logic more easily • No need to handle shared resources Copyright © GREE, Inc. All Rights Reserved. • Race condition, dead-lock • Logic separation • Parent-Child • Fault tolerance 33/56
  35. 35. Copyright © GREE, Inc. All Rights Reserved. Example of Akka 34/56
  36. 36. Copyright © GREE, Inc. All Rights Reserved. Example of Akka Match by Event type 35/56
  37. 37. Copyright © GREE, Inc. All Rights Reserved. Example of Akka Send copy to child 36/56
  38. 38. Copyright © GREE, Inc. All Rights Reserved. Example of Akka log 37/56
  39. 39. Copyright © GREE, Inc. All Rights Reserved. Stream Server Supply events to connected users • Connect by streaming • Increase connections as much as possible • Keep users data in the memory to reduce I/O • Asynchronous I/O by Finagle, Akka • Task division by Akka • Supply events to users • Update users data • Send KeepAlive to users 38/56
  40. 40. Copyright © GREE, Inc. All Rights Reserved. Summary: Architecture • Each server is separated by Queue • API, EventBus, Stream Server • Framework is selected by considering asynchronous processing • Finagle, Akka • Task is divided by Akka • Easy to write concurrent logic 39/56
  41. 41. Copyright © GREE, Inc. All Rights Reserved. Outline • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary
  42. 42. Copyright © GREE, Inc. All Rights Reserved. Obstacles 1.JVM • Full GC 2. Self-created Scala library • Sharding • ID Architecture 41/56
  43. 43. Full GC Problems Copyright © GREE, Inc. All Rights Reserved.
  44. 44. • Scala runs on JVM • We want to prevent “stop the world” • GC in young generation is faster than full GC • Try not to let it store object in old generation Copyright © GREE, Inc. All Rights Reserved. Full GC Young generation Old generation 43/56
  45. 45. Copyright © GREE, Inc. All Rights Reserved. Causes of Uncollected Reference Problem • Use method that has side effects • Reuse object everywhere • Forget to release used resources 44/56
  46. 46. Copyright © GREE, Inc. All Rights Reserved. Causes of Uncollected Reference Problem • Use method that has side effects • Reuse object everywhere • Forget to release used resources Difficult to recognize by developers 45/56
  47. 47. Copyright © GREE, Inc. All Rights Reserved. Short-Lived Object • Use val variable (final variable) • Don’t use mutable variable • Don’t leave references to objects for too long Create new objects instead of updating old ones 46/56
  48. 48. Sharding: Scalable Storage 47/56 Copyright © GREE, Inc. All Rights Reserved.
  49. 49. Copyright © GREE, Inc. All Rights Reserved. Scale up? 48/56
  50. 50. Copyright © GREE, Inc. All Rights Reserved. Scale out! 49/56
  51. 51. Copyright © GREE, Inc. All Rights Reserved. Horizontal Partitioning • Sharding is necessary! • Considering capacity and #accesses • We have Cascade, a sharding library • For PHP, not Scala • So, we created Aurora and make it OSS • Aurora and application were developed simultaneously • So difficult to integrate them during development • Code modification must be done in many parts 50/56
  52. 52. ID Architecture: UUID and InnoDB Ref:MySQL InnoDB Primary Key Choice: GUID/UUID vs Integer Insert Performance http://kccoder.com/mysql/uuid-vs-int-insert-performance/ Copyright © GREE, Inc. All Rights Reserved.
  53. 53. Ref:MySQL InnoDB Primary Key Choice: GUID/UUID vs Integer Insert Performance http://kccoder.com/mysql/uuid-vs-int-insert-performance/ Copyright © GREE, Inc. All Rights Reserved. UUID AUTO_INCREMENT #record #time
  54. 54. Drop UUID, Take Time-Based • At first, we used UUID as primary key • But this caused MySQL to insert randomly • Created a library to generate ID instead • Referred to Snowflake by Twitter • Sequential insertion based on sorted created time 53/56 Copyright © GREE, Inc. All Rights Reserved.
  55. 55. Copyright © GREE, Inc. All Rights Reserved. Summary: Obstacles & Workaround • JVM • Use short-lived objects to avoid Full GC • Self-created Scala library • Sharding • Made Scala library OSS • Integration of library to application was difficult • ID Architecture • Adopt time-based ID for sequential insertion 54/56
  56. 56. Copyright © GREE, Inc. All Rights Reserved. Outline • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary
  57. 57. Copyright © GREE, Inc. All Rights Reserved. Summary • Reasons for selecting Scala • Compatible with concurrent programming • Plan Scala learning within the team • Learning cost is expensive • Architecture & Frameworks • Each server is separated by Queue • Finagle, Akka • Shortage of libraries • We must have made libraries by ourselves • Needed to pull request to OSS repository 56/56
  58. 58. Copyright © GREE, Inc. All Rights Reserved. Thanks! @tomoyoshi_ogura, @kyo_ago, @takc923 @yoshie_777, @le_chang, @beketa, @j5ik2o and ScalaMatsuri staffs

×