Avito is the third biggest classified site in the world after Craigslit and 58.com from China. Avito nowadays is not a monolite project, but comprises dozens specialized vertical sites and applications. The introduction of microservice architecture in Avito spawned hundreds of new services. In this situation is is critical to implement common BI infrastructure, able to collect, process, combine and analyse data from all those microservices and persistent to constant changes. Avito Analytics is based on HP Vertica MPP database, highly normalized data lake and an asynchronous event bus. Those tools give Avito the ability to use all types of Machine Learning and Reporting tools, manage sites, applications and microservices. Avito is the Russian OLX. Moreover, nowadays, Avito and OLX are both part of the Naspers group, we do the same business in different counties and share experience.

1. FEBRUARY 9, 2017, WARSAW
Scalable Analytics for
Microservices Architecture
Nikolay Golov

2. FEBRUARY 9, 2017, WARSAW
Part 1 – Avito story

3. FEBRUARY 9, 2017, WARSAW
What is Avito?

4. FEBRUARY 9, 2017, WARSAW

5. FEBRUARY 9, 2017, WARSAW
Clear №1 in Russia
One Country, Many Cities, Five Verticals
Size Relative to #2 Classified Site in Key Verticals by Page Views
Moscow
St. Petersburg
Novosibirsk
N. Novgorod
Kazan
Samara
Rostov
Volgograd
Voronezh
Ufa
Chelyabinsk
Omsk
Krasnoyarsk
Vladivostok
Yakutsk
Irkutsk
Khabarovsk
1.6x2
Size of Russian
classifieds market
More than 60%
Page Views from
Mobile
40%(3)
of all used cars sold in
Russia in 2014 sold on
Avito
6 mln
listers per day, of which
ca. 10% list for the first
time
9.5bn
Page Views in Dec-2016
across all devices generated
by 40 mln people
Ca. 11,000+
Local SMEs on subscription
model
31mln
Active Items on the site as of
2016 YE
RUB32bln
2016 Revenue(1)
(+200% to 2015)
Auto
2.0x
Real Estate
15.8x
Jobs
1.5x
Services
75.8x
All Categories 4.8x
General
75.1x
✓ By far the largest classifieds in
Russia and one of the biggest
general classifieds worldwide
✓ Top-of-mind in many categories
✓ Ca. 1200 employees, of which
ca.300 moderators
✓ Profitable since Q2 2013

6. FEBRUARY 9, 2017, WARSAW
Avito now.
Source: Google Analytics, LiveInternet, Internal data
Avito Business Development
0
400
800
1 200
1 600
2 000
2 400
2 800
Jan-09 Jul-09 Jan-10 Jul-10 Jan-11 Jul-11 Jan-12 Jul-12 Jan-13 Jul-13 Jan-14 Jul-14 Jan-15 Jul-15 Jan-16
Weekly Page Views (m)
Q1 2010
Focus on Moscow
and St.Pete
September 2010
Target 13 additional cities
August 2011
Target total
of 28 cities
Q2 2013
Merger with Slando and Olx
reaffirmed #1 position in
the Russian market
+Vertical
+Listing Fees
+Pro tools
Goods C2C
+RE & Cars
+B2C
+Jobs
+Services
Path from Investment Stage to Cash Flow Generation
Stage 2Stage 1 Stage 3
Position • Competing with others • Ahead of competition • x times ahead of competition
• Heavy investment • Approximately break-even • High EBITDA marginEconomics
• Build user base • Develop business model and build leading brand • Focus on monetization enhancement; attract professional classifieds
market spend
Focus
January 2012
Avito has national
coverage
Q2 2014
Launch of Domofond, a dedicated
real estate classified
Q4 2014
Launch of a new revenue
stream: Listing Fees
Q4 2015
Sold to Naspers

7. FEBRUARY 9, 2017, WARSAW
Part 2: Let’s start from the beginning. What
issues could you face?
0
400
800
1 200
1 600
2 000
2 400
2 800
Jan-09 Jul-09 Jan-10 Jul-10 Jan-11 Jul-11 Jan-12 Jul-12 Jan-13 Jul-13 Jan-14 Jul-14 Jan-15 Jul-15 Jan-16
?? ?

8. FEBRUARY 9, 2017, WARSAW
At the beginning you do not know:
▪… your future traffic
▪… all your future data sources
▪… your future data monetization tools

9. FEBRUARY 9, 2017, WARSAW
At the beginning you hope that:
▪… your traffic will grow
▪… you will connect more and more data sources
▪… you will launch more and more data monetization
initiatives

10. FEBRUARY 9, 2017, WARSAW
Alternatives…
Data Lake
• Data in a natural state
• Hadoop based
• Schemaless
Our choice
• Full SQL support
• Single data model
• Normalization of all incoming
data

11. FEBRUARY 9, 2017, WARSAW
SCHEMA
ON WRITE
Data Lake
SCHEMA
ON READ

12. FEBRUARY 9, 2017, WARSAW
SCHEMA
ON WRITE
Data Lake
SCHEMA
ON READ
SCHEMA
ON READ
SCHEMA
ON READ

13. FEBRUARY 9, 2017, WARSAW
Back office
Click stream
BI Team
Antifraud
MDM
CRM

14. FEBRUARY 9, 2017, WARSAW

15. FEBRUARY 9, 2017, WARSAW
15 Operational DB Clickstream
• ~1 bln. Actions/day
• loading each 15 minutes
• All changes
• Hourly loading
ClickStream data
~ 90 tables
~ 190 billion records
Historized data
~ 1000 tables
~ up to 2 billion records
Daily sync
Live connection
Analysts

16. FEBRUARY 9, 2017, WARSAW
Avito DWH evolution
4
10
15
17
0
2
4
6
8
10
12
14
16
18
Cluster(s) Size
(servers)
2013 2014 2015 2016
11
26
51
76
0
10
20
30
40
50
60
70
80
Cluster(s) Size
(TB)
2013 2014 2015 2016
300
560
740
1000
0
200
400
600
800
1 000
1 200
ClickStream size
(Mln events/day)
2013 2014 2015 2016
3
14
23
29
0
5
10
15
20
25
30
35
Integrated
systems count
2013 2014 2015 2016

17. FEBRUARY 9, 2017, WARSAW
Part 3: Microservices – new level of
complexity

18. FEBRUARY 9, 2017, WARSAW
Microservices

19. FEBRUARY 9, 2017, WARSAW
Microservices – Polyglot Persistence
Registration
Search
Notifications
Billing
Moderation
Search
Search

20. FEBRUARY 9, 2017, WARSAW
ESP – Event Stream Processing
Registration
Search
Notifications
Billing
Moderation
Search
Search
ESP
ESP:
Kafka
NSQ
…

21. FEBRUARY 9, 2017, WARSAW
Click stream automation
Registration
Search
Notifications
Billing
Moderation
Search
Search
ESP
Event register:
Event 01
Event 02
New Event-> Service->Attributes
logging
expanding(?)
new chart (?)

22. FEBRUARY 9, 2017, WARSAW
Part 4: Proof, that approach works

23. FEBRUARY 9, 2017, WARSAW
Expanding the system
141
195 199
261 274 295 326
423
482
538 580
671 704
752 797 826
876 921 949 972 979 986
1032 1043 1052 1093 1120 1132 1156 1171 1205 1237 1277
1331
1496
1760
1938
2149
2223 2266
29 31 31 32 34 38 45 65 78 85 89 95 101 108 115 119 124 131 136 139 139 139 144 146 146 156 159 161 168 168 174 176 181 196 222
276 306 329 334 342
39 44 44 46 49 55 66 89 108 126 148 150 162 174 191 203 213 221 231 237 238 238 253 256 260 275 279 283 292 299 310 314 327 343
392
465
519 556 569 580
73
120 124
183 191 202 215
269 296 327 343
426 441 470 491 504 539 569 582 596 602 609 635 641 646 662 682 688 696 704 721 747 769 792
882
1019
1113
1264
1320 1344
0
500
1000
1500
2000
2500
cumulative
tables
cumulative
anchors
cumulative
ties
cumulative
attributes

24. FEBRUARY 9, 2017, WARSAW
Avito DWH – playground for analysts
0
20
40
60
80
100
120
140
160
180
0
2 000 000 000
4 000 000 000
6 000 000 000
8 000 000 000
10 000 000 000
12 000 000 000
2016-01-19 2016-02-19 2016-03-19 2016-04-19 2016-05-19 2016-06-19 2016-07-19 2016-08-19 2016-09-19 2016-10-19 2016-11-19 2016-12-19
row_count marts_count

25. FEBRUARY 9, 2017, WARSAW
Avito DWH evolution
4
10
15
17
0
2
4
6
8
10
12
14
16
18
Cluster(s) Size
(servers)
2013 2014 2015 2016
11
26
51
76
0
10
20
30
40
50
60
70
80
Cluster(s) Size
(TB)
2013 2014 2015 2016
300
560
740
1000
0
200
400
600
800
1 000
1 200
ClickStream size
(Mln events/day)
2013 2014 2015 2016
3
14
23
29
0
5
10
15
20
25
30
35
Integrated
systems count
2013 2014 2015 2016

26. FEBRUARY 9, 2017, WARSAW
Thank you!