This document describes a streaming event matching system used at Yandex. The system allows up to 500,000 subscriptions to receive notifications with low latency when matching events occur. A naive approach is discussed that would have high latency and load issues. A more sophisticated architecture is proposed using persistent data structures, Akka actors for distribution, and Zookeeper for coordination. The system has been implemented in Scala and supports over 300,000 subscriptions in production with good performance and scalability.

1. Streaming event matching
Dmitry Schitinin

2. About
https://stat.yandex.ru 2

3. About
Backend infrastructure team
› Services
› Libraries
› Frameworks
https://stat.yandex.ru 2

4. About
Backend infrastructure team
› Services
› Libraries
› Frameworks
@ Yandex.Classifieds
› auto.ru
› auto.yandex.ru
› rabota.yandex.ru
› realty.yandex.ru
› travel.yandex.ru
https://stat.yandex.ru 2

5. Introduction

6. Audience
› Software engineers/architects
› Event stream processing
› Distributed systems
4

10. Features
8

11. Features
› Stream of offers (tens per second)
8

12. Features
› Stream of offers (tens per second)
› Up to 500K subscriptions
8

13. Features
› Stream of offers (tens per second)
› Up to 500K subscriptions
› Complex multi field conditions
8

14. Features
› Stream of offers (tens per second)
› Up to 500K subscriptions
› Complex multi field conditions
› Low latency (seconds)
8

15. Motivation

16. Customer subscriptions
10

17. Customer subscriptions
› Short notification delays (minutes)
10

18. Customer subscriptions
› Short notification delays (minutes)
› Instant subscription modifications
10

19. Customer subscriptions
› Short notification delays (minutes)
› Instant subscription modifications
› Notifications transport
› email, SMS, push, pigeons, owls, etc.
10

20. Non-functional requirements
11

21. Non-functional requirements
› Scalability by subscriptions
› millions
11

22. Non-functional requirements
› Scalability by subscriptions
› millions
› Scalability by events
› tens-hundreds per second
11

23. Non-functional requirements
› Scalability by subscriptions
› millions
› Scalability by events
› tens-hundreds per second
› Fault tolerance
› nodes, network, data centers
11

31. Naive solution
Pros:
› Simplicity
› Working
19

32. Naive solution
Pros:
› Simplicity
› Working
Cons:
19

33. Naive solution
Pros:
› Simplicity
› Working
Cons:
› High notification
latency (hours, days)
19

34. Naive solution
Pros:
› Simplicity
› Working
Cons:
› High notification
latency (hours, days)
› Search system
extra load
19

35. Naive solution
Pros:
› Simplicity
› Working
Cons:
› High notification
latency (hours, days)
› Search system
extra load
› Scalability?
19

38. Not so naive approach
22

39. Not so naive approach
Pros:
22

40. Not so naive approach
Pros:
› Low latency
22

41. Not so naive approach
Pros:
› Low latency
› No extra work
22

42. Not so naive approach
Pros:
› Low latency
› No extra work
› Possibly scalable
22

43. Not so naive approach
Pros:
› Low latency
› No extra work
› Possibly scalable
Cons:
22

44. Not so naive approach
Pros:
› Low latency
› No extra work
› Possibly scalable
Cons:
› More complex
22

45. Not so naive approach
Pros:
› Low latency
› No extra work
› Possibly scalable
Cons:
› More complex, but
not complicated
22

46. Matching

48. Data model
› Event:
› Subscription:
mark=FORD&model=FOCUS&model=MONDEO&year=2015
25

50. A better way
27

51. A better way
Commonalities:
› mark=FORD&model=FOCUS
› mark=FORD&year=2010
› year=2010&price_from=500000
27

52. A better way
Commonalities:
› mark=FORD&model=FOCUS
› mark=FORD&year=2010
› year=2010&price_from=500000
Data structures
› BDD
› DAG
› Search tree
› Indices
27

65. Architecture

69. Architecture
Matcher

76. Resilience
51

77. Resilience
› Subscriptions => Tokens
51

78. Resilience
› Subscriptions => Tokens
› Token ≈responsibility
51

79. Resilience
› Subscriptions => Tokens
› Token ≈responsibility
› Nodes distribute tokens
51

87. Architecture
Notifier

95. Implementation

96. Scala
1
http://www.scalacheck.org
68

97. Scala
› Functional & concise
1
http://www.scalacheck.org
68

98. Scala
› Functional & concise
› Persistent data structures for matching tree
(1Kloc)
1
http://www.scalacheck.org
68

99. Scala
› Functional & concise
› Persistent data structures for matching tree
(1Kloc)
› No need for synchronization
1
http://www.scalacheck.org
68

100. Scala
› Functional & concise
› Persistent data structures for matching tree
(1Kloc)
› No need for synchronization
› Property-based testing (matching algorithm)
1
http://www.scalacheck.org
68

101. Akka
69

102. Akka
› actors (events)
69

103. Akka
› actors (events)
› akka-remoting (networking)
69

104. Akka
› actors (events)
› akka-remoting (networking)
› protobuf (serialization)
69

105. Akka
› actors (events)
› akka-remoting (networking)
› protobuf (serialization)
› akka-fsm (aggregation)
69

106. Zookeeper
› Exclusive token
acquirement
› Service discovery
http://zookeeper.apache.org 70

107. Conclusion

108. Achievements
› Low customer notification latency (seconds)
› No extra load
› All on 3 nodes x 2 DC
72

109. Achievements
› Low customer notification latency (seconds)
› No extra load
› All on 3 nodes x 2 DC
› 200K subs — 1G heap
72

110. 300K subscriptions in production
73

111. 300K subscriptions stress testing (cluster)
74

112. 150K subscriptions stress testing (node)
75

113. Contacts
Dmitry Schitinin
Backend infrastructure team @ Yandex.Classifieds
dimas@yandex-team.ru
76