Выступление Сергея Жемжицкого, CleverData

Company Profile
Сегментация пользователей
в online-рекламе
Spark vs Hadoop
Сергей Жемжицкий,
CTO, CleverDATA,
22 мая, 2015

cleverdata.ru | info@cleverdata.ru
International market
business development
since 2012
One of three leading IT companies in Russia
43 branches in Russia and abroad
+5500 employees
100K projects for 10K customers
Data management innovative
platform (Data Exchange Service)
Cloud Service
In-house development
Internet advertising solutions
Data Management Platforms
Customers Base Management
Web Analytics
Marketing automation
Big Data
Data Mining
Digital Intelligence
Operational Intelligence
Low Latency and NoSQL
Cloud Computing

Агенда
• Про задачу;
• Hadoop vs. Spark;
• Особенности;
• Что дальше.

publishers
AD NETWORK
AD NETWORK
AD NETWORK
AD NETWORK
AD NETWORK
AD NETWORK
advertisers
D
S
P
S
S
P
Real Time Bidding (RTB)

TRACKING DATA
publishers
COOKIE SYNCs
ACCESS LOGS
PARTNER’S DATA
3rd PARTY DATA
CLICK STREAMS
advertisers
S
S
P
D
S
P
DMP
Data Management Platform (DMP)

3rd party
data
Relational Data Store
raw data3rd party
data
3rd party
data
Raw Data Store & Processing
RealTime Data Store
user profilesaggregates
Типовые потоки данных

Типовые потоки данных :: RTB
3rd party
data
RTB
SRV
Exchange
SSP
bid req.
bid resp.
pixels :: impressions :: clicks
bid requests
user profiles
raw data3rd party
data
3rd party
data
RealTime Data Store

1st-party data
3rd party
data
RTB
SRV
Exchange
SSP
bid req.
bid resp.
pixels :: impressions :: clicks
bid requests
user profiles
raw data3rd party
data
3rd party
data
RealTime Data Store

1st-party data
• Зачем монетизировать?
• Как монетизировать?
• Чем монетизировать?

Зачем монетизировать?
Найти всех пользователей, которые
участвовали в рекламной кампании “Star Wars” [и]
видели один из баннеров “Darth Vader” или “Luke Skywalker”
в течении последних 6 дней [и]
кликнули на этот баннер [и]
посетили страницу покупки светового меча Darth’а Vader’а [и]
но так ничего и не купили
Для того, чтобы
сделать ретаргетинг персонифицированным баннером со
скидкой на меч в 40%

find all users who have
taken part in campaign[s] “Star Wars” [and]
viewed banner[s] “Darth Vader” or “Luke Skywalker”
during [last] 6 day[s] [and]
clicked banner[s] “Darth Vader's lightsaber” [and]
visited buying area of “Darth Vader's lightsaber” [and]
not visited order confirmed area of “Darth Vader's lightsaber”
Как монетизировать?
[impression]
[click]
[tr. pixel]
[tr. pixel]
id cookie event_id event_type campaign_id timestamp …
1 c1 “Darth Vader” impression “Star Wars” 2015-04-20 14:25:11.462 …
2 c1 “Darth Vader's lightsaber” click “Star Wars” 2015-04-21 06:31:12.157 …
3 c1 “Darth Vader's lightsaber” tr. pixel “Star Wars” 2015-04-22 18:57:19.628 …
[cookies]

Как монетизировать?
reducefind all users who have
taken part in campaign[s] “Star Wars”
viewed banner[s] “Darth Vader” or
“Luke Skywalker” during [last] 6 day[s]
clicked banner[s] “Darth Vader's
lightsaber”
visited buying area of “Darth Vader's
lightsaber”
not visited order confirmed area of “Darth
Vader's lightsaber”
(c1, 0)
(c1, 1)
(c1, 2)
(c1, 3)
Ø
map
(c1, 0;1;2;3)
true(0) and
true(1) and
true(2) and
true(3) and
not false(4)
C1

VS.

MR vs Spark :: Правда жизни
• Стильно;
• Модно;
• Молодежно.

Spark :: Размер

Перед тем, как смотреть на Hadoop

Map-Reduce :: Размер

Материалы и инструменты
Hardware (3 Nodes)
• 12 Core AMD Opteron™ 6338P
~ 2.8 GHz
• 64 GB RAM
• 1 GBPS NICs
Software
• CDH 5.3.1 (Hadoop 2.5.0)
• Spark 1.2.0
Data
• 14.2 GB of raw data
• 61.1 M of transactions
• 128 MB block size

MR vs Spark :: Время выполнения

Spark :: Exec-cores vs Num-execs

MR vs Spark :: Инициализация
MR
protected void setup(Context ctx)
o.a.h.c.Configured
distributed cache
Spark
mapRegion
broadcast vars

MR vs Spark :: Параллелизм
MR
mapred.reduce.tasks
mapreduce.job.reduces
splittable formats
Spark
spark.default.parallelism
num-executors, executor-cores in
yarn
numTasks в groupByKey,
reduceByKey, aggregateByKey…

MR vs Spark :: Зависимости
MR
o.a.h.u.Tool
o.a.h.u.ToolRunner
-conf app.conf
-files
-libjars
setUserClassesTakesPrecedence
Spark
--jars
--files
--conf
--driver-java-options
spark.driver.extraJavaOptions
spark.executor.extraJavaOptions
spark.driver.userClassPathFirst
spark.executor.userClassPathFirst

MR vs Spark :: Secondary Sort
MR
setSortComparatorClass
setGroupingComparatorClass
setPartitionerClass
Spark
repartitionAndSortWithinPartitions
mapPartitions
Entire partition processing result
must be able to fit in memory

MR vs Spark :: Тестирование
MR
MRUnit
o.a.h.h.MiniDFSCluster
o.a.h.m.MiniMRCluster
o.a.h.y.s.MiniYARNCluster
o.a.h.m.v2.MiniMRYarnCluster
Spark
Local executor

Что дальше и почему Spark?
• Spark Streaming;
• Micro Batches;
• λ-архитектура.
без серьезного хирургического вмешательства

Спасибо за вопросы!

info@cleverleaf.co.uk :: info@cleverdata.ru
cleverleaf.co.uk :: cleverdata.ru
1dmp.io :: crawler.1dmp.io
facebook.com/CleverData :: +7 (495) 967-66-50

Выступление Сергея Жемжицкого, CleverData

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (15)

Similar to Выступление Сергея Жемжицкого, CleverData

Similar to Выступление Сергея Жемжицкого, CleverData (20)

Recently uploaded

Recently uploaded (20)

Выступление Сергея Жемжицкого, CleverData