Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Distributed ID generator in ChatWork

1,122 views

Published on

チャットワークのメッセージシステムを支える新分散ID発行器の内部

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Distributed ID generator in ChatWork

  1. 1. チャットワークのメッセージシステム を支える 新分散ID発行器の内部 安田裕介/Yusuke Yasuda (@TanUkkii007) Distributed ID generator in ChatWork
  2. 2. 2018/02/27 © ChatWork All rights reserved. Agenda ● What is our motivation for distributed ID ● How our distributed ID generator works ● Designing distributed ID generator with actor model ● Implementing distributed ID generator with actor model ● Distributed ID generator in the wild
  3. 3. 2018/02/27 © ChatWork All rights reserved. About me ● Yusuke Yasuda / 安田裕介 ● @TanUkkii007 ● Working for ChatWork for 2 years ● Scala developer
  4. 4. 2018/02/27 © ChatWork All rights reserved. About ChatWork
  5. 5. 2018/02/27 © ChatWork All rights reserved. What is our motivation for distributed ID
  6. 6. 2018/02/27 © ChatWork All rights reserved. Messaging system architecture overview You can find more information about our architecture at Kafka summit 2017. Today’s topic
  7. 7. 2018/02/27 © ChatWork All rights reserved. Motivation ● Migration from MySQL to Kafka/HBase ○ High scalability ○ No single point of failure ● compatibility with existing IDs ○ Time-ordered, sortable integer IDs ● ID space extension ○ 32bit → 64bit ● ID generator itself is required to be scalable and distributed
  8. 8. 2018/02/27 © ChatWork All rights reserved. Snowflake ● Distributed ID generator developed by Twitter ○ https://github.com/twitter/snowflake ● Motivation: migration to Cassandra from MySQL ● Roughly time-ordered, sortable 64bit ID ● > 10k ID/s per process, ~ 2ms response rate 41 bit timestamp 5 bit datacenter ID 5 bit worker ID 12 bit sequenceNr … … … …
  9. 9. 2018/02/27 © ChatWork All rights reserved. ZooKeeper ● Developed by Yahoo! Research ○ to simplify distributed system implementation by supporting common patterns used in distributed systems ● Quorum based coordination ● Total order broadcast ● Filesystem-like API ○ Create, GetData, SetData, Exists, GetChildren, Delete ○ Watching node changes We use ZooKeeper to coordinate worker IDs in distributed ID generator.
  10. 10. 2018/02/27 © ChatWork All rights reserved. Distributed ID generator system overview ID worker ID worker ID worker ZooKeeper ID client ZooKeeper ensemble ZooKeeper ZooKeeper ID generator ID client in Message Write API
  11. 11. 2018/02/27 © ChatWork All rights reserved. How our distributed ID generator works
  12. 12. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 Command: GetChildren Path: /id-worker/1 Watch: True Router Datacenter 1 root node command/event watch
  13. 13. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker Command: GetChildren Path: /id-worker/1 Watch: True Router command/event watch
  14. 14. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** Command: Create Path: /id-worker/1/1 Data: akka.tcp://system@11.2.9.14:*** mode: Ephemeral Router command/event watch
  15. 15. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** Event: NodeChildrenChanged Path: /id-worker/1 Router command/event watch
  16. 16. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** Command: GetChildren Path: /id-worker/1 Watch: True Router command/event watch
  17. 17. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** Command: GetData Path: /id-worker/1/1 Watch: False Router command/event watch
  18. 18. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** Identify Router command/event watch
  19. 19. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** ActorIdentity worker1 Router command/event watch
  20. 20. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** ID request/response worker1 Router command/event watch
  21. 21. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** ID worker Command: GetChildren Path: /id-worker/1 Watch: True worker1 Router command/event watch
  22. 22. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** ID worker /id-worker/1/2 akka.tcp://system@11.2.9.15:*** Command: Create Path: /id-worker/1/2 Data: akka.tcp://system@11.2.9.15:*** mode: Ephemeral worker1 Router command/event watch
  23. 23. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** ID worker /id-worker/1/2 akka.tcp://system@11.2.9.15:*** Event: NodeChildrenChanged Path: /id-worker/1 worker1 Router command/event watch
  24. 24. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** ID worker /id-worker/1/2 akka.tcp://system@11.2.9.15:*** Command: GetChildren Path: /id-worker/1 Watch: True worker1 Router command/event watch
  25. 25. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** ID worker /id-worker/1/2 akka.tcp://system@11.2.9.15:*** Command: GetData Path: /id-worker/1/2 Watch: False worker1 Router command/event watch
  26. 26. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** ID worker /id-worker/1/2 akka.tcp://system@11.2.9.15:*** Identify worker1 Router command/event watch
  27. 27. ID worker discovery via ZooKeeper ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** worker1ID worker /id-worker/1/2 akka.tcp://system@11.2.9.15:*** ActorIdentity worker2 Router command/event watch
  28. 28. Client-side failure handling ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** worker1ID worker /id-worker/1/2 akka.tcp://system@11.2.9.15:*** worker2 Router command/event watch
  29. 29. Client-side failure handling ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** worker1ID worker worker2 Router Event: NodeChildrenChanged Path: /id-worker/1 command/event watch
  30. 30. Client-side failure handling ZooKeeper ID client /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** worker1ID worker Router Command: GetChildren Path: /id-worker/1 Watch: True command/event watch
  31. 31. Worker ID Consensus process 1. No worker ID duplication invariant ZooKeeper /id-worker/1 ID worker Command: GetChildren Path: /id-worker/1 Watch: True command/event watch
  32. 32. ZooKeeper /id-worker/1 ID worker ID worker Command: GetChildren Path: /id-worker/1 Watch: True Worker ID Consensus process 1. No worker ID duplication invariant command/event watch
  33. 33. ZooKeeper /id-worker/1 ID worker /id-worker/1/1 akka.tcp://system@11.2.9.14:*** Command: Create Path: /id-worker/1/1 Data: akka.tcp://system@11.2.9.14:*** mode: Ephemeral ID worker Failed with NODEEXISTS Worker ID Consensus process 1. No worker ID duplication invariant command/event watch
  34. 34. ZooKeeper /id-worker/1 ID worker /id-worker/1/0 akka.tcp://system@11.2.9.14:*** ID worker /id-worker/1/1 akka.tcp://system@11.2.9.15:*** akka.tcp://system@11.2.9.16:***/id-worker/1/31 ID worker ID worker Command: GetChildren Path: /id-worker/1 Watch: True Worker ID Consensus process 2. No out of range worker ID creation command/event watch
  35. 35. ZooKeeper /id-worker/1 ID worker /id-worker/1/0 akka.tcp://system@11.2.9.14:*** ID worker /id-worker/1/1 akka.tcp://system@11.2.9.15:*** akka.tcp://system@11.2.9.16:***/id-worker/1/31 ID worker ID worker Worker ID Consensus process 2. No out of range worker ID creation Suspended because no available worker IDs. command/event watch
  36. 36. ZooKeeper /id-worker/1 ID worker ID worker /id-worker/1/1 akka.tcp://system@11.2.9.15:*** akka.tcp://system@11.2.9.16:***/id-worker/1/31 ID worker ID worker Worker ID Consensus process 2. No out of range worker ID creation command/event watch
  37. 37. ZooKeeper /id-worker/1 ID worker ID worker /id-worker/1/1 akka.tcp://system@11.2.9.15:*** akka.tcp://system@11.2.9.16:***/id-worker/1/31 ID worker ID worker Worker ID Consensus process 2. No out of range worker ID creation Event: NodeChildrenChanged Path: /id-worker/1 command/event watch
  38. 38. ZooKeeper /id-worker/1 ID worker ID worker /id-worker/1/1 akka.tcp://system@11.2.9.15:*** akka.tcp://system@11.2.9.16:***/id-worker/1/31 ID worker ID worker Worker ID Consensus process 2. No out of range worker ID creation Command: GetChildren Path: /id-worker/1 Watch: True command/event watch
  39. 39. ZooKeeper /id-worker/1 ID worker ID worker /id-worker/1/1 akka.tcp://system@11.2.9.15:*** akka.tcp://system@11.2.9.16:***/id-worker/1/31 ID worker ID worker Command: Create Path: /id-worker/1/1 Data: akka.tcp://system@11.2.9.14:*** mode: Ephemeral /id-worker/1/0 akka.tcp://system@11.2.9.14:*** Worker ID Consensus process 2. No out of range worker ID creation command/event watch
  40. 40. 2018/02/27 © ChatWork All rights reserved. Automaton notation of ID worker Create my worker ID Initial state Check available worker IDs Wait until a worker ID gets available Ready to generate message IDs Fetch list of worker IDsStart Create a worker ID node Retry Retry Something happened to my worker ID Retry Worker ID conflict No available worker IDs Retry
  41. 41. 2018/02/27 © ChatWork All rights reserved. Designing distributed ID generator with actor model
  42. 42. 2018/02/27 © ChatWork All rights reserved. Processes, Modules, Automata, Steps Process 1 Module 1 Module 2 Module 3 Process 2 Module 1 Module 2 Module 3 Automaton local interaction communication
  43. 43. 2018/02/27 © ChatWork All rights reserved. Distributed algorithm abstraction ● Process: the unit of failure, the unit of communication ● Module: building block of processes ○ communication with modules on peer process ○ local interaction with modules on the same process ● Automata: set of states and transitions that regulates computation steps ● Step: distributed algorithm consists of sequence of steps ○ e.g. receiving a message, sending a message, executing a local computation syntactically identical but different notion Reliable and Secure Distributed Programming 2nd ed.
  44. 44. 2018/02/27 © ChatWork All rights reserved. Algorithm abstraction to concrete implementation mapping ● Process → Actor (top level actor with remote interface) ● Module → Actor ○ communication → message passing to remote actors ○ local interaction → message passing to local actors ● Layers of modules → Actor hierarchy tree ● Automata ○ State → Receive partial functions and its internal states ○ Transition → context.become() ● Step: ○ receiving a message → case clauses of Receive function ○ sending a message → ! or tell() function ○ executing a local computation: arbitrary computation syntactically identical
  45. 45. 2018/02/27 © ChatWork All rights reserved. Modules in ID worker process ● IdGenerator: exports communication interface ● IdWorker: calculates Snowflake ID ● ZNodeMaster: manages worker ID ZNode ● ReactiveZookeeper: Actor based ZooKeeper client ID client process ZooKeeper process IdGenerator IdWorker ZNodeMaster ReactiveZookeeper ID worker process
  46. 46. 2018/02/27 © ChatWork All rights reserved. IdWorker IdGenerator ReactiveZookeeper Designing actor hierarchy based on message flow simplifies its implementation. See Akka in Action section 4.3. Mapping layers of modules to actor hierarchy based on message flow. ZNodeMaster IdGenerator IdWorker ZNodeMaster ReactiveZookeeper ID worker process message flow parent child relationship Designing actor hierarchy based on message flow
  47. 47. 2018/02/27 © ChatWork All rights reserved. Designing actor hierarchy based on failure handling IdWorker IdGenerator ReactiveZookeeper ZNodeMaster ZooKeeper application is only valid within a session. Instead of handling session timeout everywhere, locate ReactiveZooKeeper at the top of the hierarchy. Just let it crash on session timeout. message flow parent child relationship IdGenerator IdWorker ZNodeMaster ReactiveZookeeper ID worker process
  48. 48. 2018/02/27 © ChatWork All rights reserved. Implementing distributed ID generator with actor model
  49. 49. 2018/02/27 © ChatWork All rights reserved. What ZooKeeper application needs ● Asynchronous ● Event driven ● Passive: Don't call us, we'll call you style ● Stateful ● Complicated state machine ● Retry everything ● Crash immediately in fatal situation (session expiration) ● Recover gracefully
  50. 50. 2018/02/27 © ChatWork All rights reserved. ReactiveZooKeeper Let-it-crash style ZooKeeper client based on Akka actor https://github.com/TanUkkii007/reactive-zookeeper
  51. 51. 2018/02/27 © ChatWork All rights reserved. Implementing automata with Akka class ZNodeMaster(reactiveZK: ActorRef) extends Actor { override def receive: Receive = initial def initial: Receive = { case Start => self ! CheckWorkerIds context.become(checkingWorkerIds) } def checkingWorkerIds: Receive = { case CheckWorkerIds => reactiveZK ! GetChildren("id-worker/1", watch = true) case ChildrenGot(path, children, _) if children.length <= maxWorkerId => val workerId = chooseId(children) context.become(creatingWorkerId(workerId)) self ! CreateWorkerId } def creatingWorkerId(workerId: Int): Receive = { case CreateWorkerId => reactiveZK ! Create(workerIdPath(workerId), address, Ids.OPEN_ACL_UNSAFE.asScala.toList, CreateMode.EPHEMERAL) } } Create my worker ID Initial state Check available worker IDs Fetch list of worker IDsStart State 1 State 2 State 3 state transition with context.become internal state as a closure variable that is only visible within the state ZNodeMaster module
  52. 52. 2018/02/27 © ChatWork All rights reserved. Implementing self-loop of automata Check available worker IDs Retry def checkingWorkerIds: Receive = { case CheckWorkerIds => reactiveZK ! GetChildren("id-worker/1", watch = true) case ChildrenGot(path, children, _) if children.length <= maxWorkerId => val workerId = chooseId(children) context.become(creatingWorkerId(workerId)) self ! CreateWorkerId case GetChildrenFailure(e, _, _) if e.code() == Code.CONNECTIONLOSS => self ! CheckWorkerIds } Retry on connection loss by sending the same message to self. ZNodeMaster module
  53. 53. 2018/02/27 © ChatWork All rights reserved. Implementing communication interface def receive: Receive = { case GetIdWorkerAddress(WorkerId(workerId)) => reactiveZK ! GetData(s"/${settings.rootNode}/${settings.dcId}/$workerId", watch = false, ctx = workerId) case DataGot(path, data, stat, workerId: Long) => val idWorkerZNode = deserializeIdWorkerZNode(data) val workerPath = idWorkerZNode.fullActorPath val selection = context.actorSelection(workerPath) selection ! Identify(workerId) case ActorIdentity(workerId: Long, Some(workerRef)) => workers += WorkerId(workerId) -> context.watch(workerRef) val newRoutee = IdWorkerRoutee(WorkerId(workerId), workerRef) if (!router.routees.contains(newRoutee)) router = router.addRoutee(newRoutee) case msg: GenerateId => router.route(IdWorkerGenerateId(msg.requestId.toString), sender()) } Remote messages. ID worker client implementation Configuring RemoteActorRef Provider let us write the communication syntactically same as the local interaction.
  54. 54. 2018/02/27 © ChatWork All rights reserved. Implementing crash recovery class ZooKeeperSessionActor(childProps: Props) extends Actor with WatcherCallback{ val zookeeper = new ZooKeeper("zookeeper:2181", 5000, watchCallback(self)) val zookeeperOperation: ActorRef = context.actorOf(ZooKeeperOperationActor.props(zookeeper)) val childActor: ActorRef = context.actorOf(childProps, childName)) def receive: Receive = { case ZooKeeperWatchEvent(e) => e.getState match { case Expired => throw ZooKeeperSessionRestartException(None) case _ => } case cmd: ZKOperations.ZKCommand => zookeeperOperation forward cmd case other => childActor forward other } override def postStop(): Unit = { zookeeper.close() super.postStop() } } Throw exception on ZooKeeper session expiration. Note that ZooKeeper application is valid only if its session is valid. Just let it crash on session expiration. Akka automatically restart child actors. Stale states of actors are refreshed on restart. ReactiveZooKeeper module
  55. 55. 2018/02/27 © ChatWork All rights reserved. Edge case that breaks ID uniqueness ZooKeeper ID worker 1 ID worker 2 ID client 1 delete worker ID 1 session expired NodeChildren Changed my worker ID is 1 my worker ID is 1 ID client 2 worker 1’s ID is 1 worker 2’s ID is 1 worker 1’s ID is 1 worker 2’s ID is 1 NodeChildren Changed network partition create worker ID 1 session timeout ID duplication risk! network recovered
  56. 56. 2018/02/27 © ChatWork All rights reserved. Mitigate condition that breaks consensus inferred by FLP impossibility def receive: Receive = { case ZooKeeperWatchEvent(e) => e.getState match { case Expired => throw ZooKeeperSessionRestartException(None) case Disconnected => throw new ConnectionRecoveryTimeoutException(connectionTimeout) case _ => } case cmd: ZKOperations.ZKCommand => zookeeperOperation forward cmd case other => childActor forward other } Fail fast on disconnection. ReactiveZooKeeper module
  57. 57. 2018/02/27 © ChatWork All rights reserved. Distributed ID generator in the wild
  58. 58. 2018/02/27 © ChatWork All rights reserved. ID generation latency and throughput The stress test used a gatling plugin for Akka Remote protocol. https://github.com/chatwork/gatling-akka Single ID worker performance
  59. 59. 2018/02/27 © ChatWork All rights reserved. Effect of ZooKeeper node down ID generation throughput The number of ID workers connected to the downed ZooKeeper Error count of ID clients # of affected workers by ZooKeeper down Effect 1/3 Not observable 2/3 Relatively high error rate 3/3 Throughput decrease High error rate We use redundant requests to mitigate effects of ID worker down.
  60. 60. 2018/02/27 © ChatWork All rights reserved. Write API latency (average) ~6ms improvement. The distributed ID generator release
  61. 61. 2018/02/27 © ChatWork All rights reserved. Write API latency (100, 95, 50 pt.) The distributed ID generator release 95 pt. is not much improved.
  62. 62. 2018/02/27 © ChatWork All rights reserved. ID Generator latency (average) The distributed ID generator release ~8ms improvement.
  63. 63. 2018/02/27 © ChatWork All rights reserved. ID Generator latency (100, 95, 50 pt.) The distributed ID generator release Worse 95pt. Possible fixes: - use higher throughput for dispatcher - use pinned dispatcher - competitive redundant requests with scatter gather pattern
  64. 64. 2018/02/27 © ChatWork All rights reserved. We are hiring! https://corp.chatwork.com/ja/recruit/

×