Samza london hug presentation Aug 5 2015 - Garry Turkington

Copyright © 2015 Improve Digital - All Rights Reserved
Example Use of Samza at
Improve Digital
Garry Turkington CTO
g.turkington@improvedigital.com
@garryturk

• Cloud-based Real Time Advertising
Technology
• Focus on the premium publisher /
media owner
• Integrations with thousands of
Demand Partners
• Decision driven by Real Time Data
• Offices in UK, NL, DE & ES
• +100 Premium Publishers
IMPROVE
DIGITAL
ABOUT US

Data Sources

• We have lots of data coming from the ad server fleet
• We need to get to it as fast as possible
• Kafka was a natural hub for that data
• And Samza was a natural extension for message
processing
• Used for both single message notifications and
aggregations
Why Samza is useful

Samza integration
• A Samza job runs within multiple YARN
containers
• Each container runs multiple tasks
• The task instances are the custom code
• Each task is the dedicated consumer of a
single partition in each of its input topics
• Samza checkpoints task progress to allow
recovery after task failure/restart

Samza API
Every Samza task implements the StreamTask
interface which has one method:
Void process(
IncomingMessageEnvelope envelope,
MessageCollector collector,
TaskCoordinator coordinator);

Samza API (continued)
Optionally the initableTask and windowableTask
interfaces can also be implemented:
void init(Config config, TaskContext context);
• Void window(MessageCollector collector,
TaskCoordinator coordinator);

For each client produce 1 minute aggregates:
• Timestamp
• Client
• Number of impressions (ads won)
• Number of bids received
• Amount of budget spent
Example Problem Statement

• Log files pushed into S3 which creates SQS
notifications
• Ingest application reads the messages, and pulls files
• Then splits files and pushes records into Kafka
• Ensures correct partition key
• Hold the obvious question about where the ingest
lives…
Step 1: Getting Data Into Kafka

S3Ad Server
Logs
Ingest App
SQS Notifications Get
Request
Logs
Kafka
Log Records
Getting Data Into Kafka

• Raw ingest rate is very high
• So a first job does 10 second aggregations on the
input
• This also does some column filtering/cleaning
• Not strictly necessary but the output proved very
useful
• Plus helps reduce YARN resource requirements
Step 2: Top-Level Aggregation

Top Level Aggregation
Impression
Reduction
Imp Logs Bid Logs
Bids
Reduction

• First step that is specific to this use case
• From the 10 second aggs combines into per-minute
aggregates
• Importantly also reduces to a much smaller set of
key fields
• Builds aggregates using persistent task state
Step 3: Main Aggregation

Main Aggregation
Impression
Reduction
Imp Logs Bid Logs
Bids
Reduction
Bids
Aggregation
Impression
Aggregation
1 min imp aggs 1 min bid aggs

• Next job in the chain creates composite structures
• The imp and bid aggregates are combined into
single records
• Heavily uses Samza persistent state
• Data object is stored for each minute and fields filled
from both streams
Step 4: Crossing the Streams

Crossing The Stream
Impression
Reduction
Imp Logs Bid Logs
Bids
Reduction
Bids
Aggregation
Impression
Aggregation
Combiner
Composite records

This doesn’t give us revenue data converted to client
currency
But to know how to convert we need reference data:
• Exchange rate information
• The preferred currency for each client
• We use Samza bootstrap streams to inject this
information
• Then update as new data arrives
Reference data

Reference Data
Impression
Reduction
Imp Logs Bid Logs
Bids
Reduction
Bids
Aggregation
Impression
Aggregation
Combiner
Composite records
Reference
Streams

Step 5: Getting Data Out of Kafka
Final job reads the composite records and pushes
downstream
Where export lives is flexible:
• Write directly from final job
• Add a specific output job
• Use an external client
• Kafka copycat may give a common framework here

Getting Data Out of Kafka
Impression
Reduction
Imp Logs Bid Logs
Bids
Reduction
Bids
Aggregation
Impression
Aggregation
Combiner
Composite records
Reference
Streams
Exporter
Downstream Data

Final Bill of Materials
To meet our requirement we use a combination
of:
• External ingest client
• Initial data reduction job
• Data aggregation job
• Stream combiner job
• Data exporter job

Summary
• Samza gives great flexibility for small window aggregation
• Decomposing into multiple jobs creates reusable intermediate
data
• Remember the output of each job is available to any Kafka
client
• Much thought often goes into getting data into and out of
Kafka
• Kafka copycat may make that a whole lot easier

Questions?

Samza london hug presentation Aug 5 2015 - Garry Turkington

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Samza london hug presentation Aug 5 2015 - Garry Turkington

Similar to Samza london hug presentation Aug 5 2015 - Garry Turkington (20)

Recently uploaded

Recently uploaded (20)

Samza london hug presentation Aug 5 2015 - Garry Turkington