Reactor pattern
Event handling pattern
Dejan Pekter
Software Development Engineer II
Nordeus and me
● Over 150 employees
● Top Eleven
○ ~ 12 000 000 MAU
○ ~ 5 000 000 DAU
○ Cross-platform pioneer
○ Most popular online sports game
● Backend Software Development Engineer
○ 250+ servers
○ 10+ types
○ 70.000 requests / minute
○ Every fail is expected
Goal
● Share knowledge on Reactor design pattern
used on our servers
● In example of server that is
○ Highly concurrent
○ Highly responsive
○ Well designed
● Consider different solutions
○ Incrementally find solution
○ Pros
○ Cons
Prerequisites
● Design pattern
● Thread
● Context switch
● Request
● Response
Easy solution
● Wait for clients to connect
● For every accepted client
○ Create thread which will take responsibility over that
client
Easy solution - conclusion
● Good:
○ Intuitive
○ Readable code
○ Easy to understand
● Bad:
○ 1000 users ~ 1000 threads
○ Memory overhead
○ Context switch overhead
○ Create/kill overhead
Improved solution
● Create thread pool with N threads
● Accept client
○ Pass client to thread pool
○ Thread pool assigns thread to that client
Improved solution - conclusion
● Good:
○ Constrained number of threads
○ Threads are reused
● Bad:
○ Harder to implement
○ Input/output is mixed with business logic
○ Possibility of starvation
Basic nuclear reactor theory
NO!
Re-act!
How would you react on
good joke (event) that I can’t really tell?
FunnyThingHandler
How would you react on
LOGIN_OR_REGISTER
You would not!
Server would.
Event granulation
CLIENT MESSAGE
Reactor pattern
Reactor
Events
Callbacks
Event handler
EventHandler
+ handleEvent()
+ getEventId()
LoginHandler
+ handleEvent()
+ getEventId()
RegisterHandler
+ handleEvent()
+ getEventId()
Reactor detailed design
EventHandler
+ handleEvent()
+ getEventId()
LoginHandler
+ handleEvent()
+ getEventId()
RegisterHandler
+ handleEvent()
+ getEventId()
Server
Reactor
+ handleEvents()
+ registerHandler()
+ removeHandler()
dispatches1 *
creates
Communicator
+ getEvents()
1
1
Reactor phases
● Initialize phase
○ Registration of event handlers
● Event handling phase
○ Gather all ready messages
○ Process all messages
■ Get requests
■ Identify event handler
■ Execute callback
Reactor - Sequence Diagram
Main Program Event Handler Reactor
Events
registerHandler()
instantiateHandler()
handleEvents()
handleEvent()
1) Initialize phase
2) Event handling phase
concreteHandler
event
Reactor solution - conclusion
● Good
○ Logic is separated
○ Easy to add new event handler
○ Granulation to message level
● Bad
○ Harder to debug
○ Not intuitive
○ Single threaded
Our improvements
● One reactor is not enough N reactors
● Separate logic even more
○ Server - single thread
■ Accept clients
■ Dispatch to reactor by round robin
○ Reactor
■ IO responsibility
■ Build application level messages
■ Find event handler
■ send to dispatcher
○ Dispatcher - thread pool
■ Execute callback
Our improvements - conclusion
● Good
○ Responsibilities are fully separated
○ 70.000 requests per minute in production
○ Tested even for order of magnitude more
○ Good design
● Bad
○ Even harder debugging
Conclusion
● Reactor pattern
○ Great maintainability
○ Forget about how messages arrive, they just do
○ Single threaded by definition
○ Multithreaded in production
● Business
○ Invest time and money in good solution
○ You get new stuff almost for free
Remember...
● Don’t reinvent the wheel
● Got the wheel, but you need a plane
○ think
○ improve
○ build it yourself
Don’t forget to...
WORK AND PLAY
Thanks for your attention!
Questions?

Dejan Pekter / Nordeus – Reactor design pattern

  • 1.
    Reactor pattern Event handlingpattern Dejan Pekter Software Development Engineer II
  • 2.
    Nordeus and me ●Over 150 employees ● Top Eleven ○ ~ 12 000 000 MAU ○ ~ 5 000 000 DAU ○ Cross-platform pioneer ○ Most popular online sports game ● Backend Software Development Engineer ○ 250+ servers ○ 10+ types ○ 70.000 requests / minute ○ Every fail is expected
  • 3.
    Goal ● Share knowledgeon Reactor design pattern used on our servers ● In example of server that is ○ Highly concurrent ○ Highly responsive ○ Well designed ● Consider different solutions ○ Incrementally find solution ○ Pros ○ Cons
  • 4.
    Prerequisites ● Design pattern ●Thread ● Context switch ● Request ● Response
  • 5.
    Easy solution ● Waitfor clients to connect ● For every accepted client ○ Create thread which will take responsibility over that client
  • 6.
    Easy solution -conclusion ● Good: ○ Intuitive ○ Readable code ○ Easy to understand ● Bad: ○ 1000 users ~ 1000 threads ○ Memory overhead ○ Context switch overhead ○ Create/kill overhead
  • 7.
    Improved solution ● Createthread pool with N threads ● Accept client ○ Pass client to thread pool ○ Thread pool assigns thread to that client
  • 8.
    Improved solution -conclusion ● Good: ○ Constrained number of threads ○ Threads are reused ● Bad: ○ Harder to implement ○ Input/output is mixed with business logic ○ Possibility of starvation
  • 9.
  • 10.
  • 11.
    How would youreact on good joke (event) that I can’t really tell?
  • 12.
  • 13.
    How would youreact on LOGIN_OR_REGISTER
  • 14.
  • 15.
  • 16.
  • 17.
    Event handler EventHandler + handleEvent() +getEventId() LoginHandler + handleEvent() + getEventId() RegisterHandler + handleEvent() + getEventId()
  • 18.
    Reactor detailed design EventHandler +handleEvent() + getEventId() LoginHandler + handleEvent() + getEventId() RegisterHandler + handleEvent() + getEventId() Server Reactor + handleEvents() + registerHandler() + removeHandler() dispatches1 * creates Communicator + getEvents() 1 1
  • 19.
    Reactor phases ● Initializephase ○ Registration of event handlers ● Event handling phase ○ Gather all ready messages ○ Process all messages ■ Get requests ■ Identify event handler ■ Execute callback
  • 20.
    Reactor - SequenceDiagram Main Program Event Handler Reactor Events registerHandler() instantiateHandler() handleEvents() handleEvent() 1) Initialize phase 2) Event handling phase concreteHandler event
  • 21.
    Reactor solution -conclusion ● Good ○ Logic is separated ○ Easy to add new event handler ○ Granulation to message level ● Bad ○ Harder to debug ○ Not intuitive ○ Single threaded
  • 22.
    Our improvements ● Onereactor is not enough N reactors ● Separate logic even more ○ Server - single thread ■ Accept clients ■ Dispatch to reactor by round robin ○ Reactor ■ IO responsibility ■ Build application level messages ■ Find event handler ■ send to dispatcher ○ Dispatcher - thread pool ■ Execute callback
  • 23.
    Our improvements -conclusion ● Good ○ Responsibilities are fully separated ○ 70.000 requests per minute in production ○ Tested even for order of magnitude more ○ Good design ● Bad ○ Even harder debugging
  • 24.
    Conclusion ● Reactor pattern ○Great maintainability ○ Forget about how messages arrive, they just do ○ Single threaded by definition ○ Multithreaded in production ● Business ○ Invest time and money in good solution ○ You get new stuff almost for free
  • 25.
    Remember... ● Don’t reinventthe wheel ● Got the wheel, but you need a plane ○ think ○ improve ○ build it yourself
  • 26.
  • 27.
    Thanks for yourattention! Questions?