This document describes how to summarize sales data by state and time period. It involves transforming customer and sales records to key-value pairs with state, joining on customer ID, and then reducing by state and time period (hourly, daily, monthly, yearly) to sum the sales amounts. The results are unioned and saved as a text file with the state, year, month, day, hour and total sales for each time period. Techniques like broadcasting the smaller customer dataset, using a partitioner based on state, and persisting datasets are discussed to reduce shuffling during the joins and reductions.

1. ABOUT ME
➤ EE/CS/CB
➤ BS EE, Robotics, Shanghai Univ
➤ MS Computational Biology,
School of Computer Science, CMU

2. ABOUT ME
Tech~
➤ Large Genome Dataset ~ Open Big
data world
➤ Interested in data and open source
➤ Familiar with Hadoop ecosystem
and various open source data tools
➤ Insight Data Engineer Fellow

3. ABOUT ME
Non-tech~
➤ Enjoy most of the indoor non-
physical activities ~
➤ Like drawing small picture stories
to record fun facts happen around
➤ Like dressing little yellow and
observing his love life

4. Search Ads Playground
Online Search Ads Platform
&
Real Time Campaign Monitoring
Grace Lu

5. MOTIVATION
➤ Every website, every application with search engine can do
search advertisement business
url：searchad.fun or http://107.23.229.58/
Ads
Ads

6. SEARCH ADVERTISEMENT RANKING
“Don’t be evil.” —— Google
How do these ads being selected?
Quality Bid Price
Relevancy P-Click
Ad
Search
Result
url：searchad.fun or http://107.23.229.58/

7. ONLINE SEARCH AD PLATFORM
➤ Quality Score = Relevancy * Click_Prediction (p-click)
➤ Final Ranking Score = Quality Score * Bid Price
Search Query
batch ETL
product data
user click log
Top ads
“black chair”
Ad Engine
url：searchad.fun or http://107.23.229.58/

8. ETL
➤ Clean raw product data - store in MySql
➤ Train word2vector - generate synonyms - store in Memcached
➤ Build inverted index in Memcached
➤ Process log file to extract features
➤ Process log file to generate regression model

9. CTR FEATURES
➤ pClick = probability of click/estimated Click Through Rate
(CTR)
➤ pClick Features extracted from search log and stored in key-
value store
log: Device IP, Device id,Session id,Query,AdId,CampaignId,Ad_category_Query_category(0/1),clicked(0/1)

10. REALTIME MONITORING
➤ Let advertiser know …
➤ Can be extend to monitor activities in other system
url：searchad.fun or http://107.23.229.58/

11. REALTIME MONITORING
➤ Log events
Ad Impression Events
Ad Click Events
Chance Events
AD Server
url：searchad.fun or http://107.23.229.58/

12. ANALYTICS DASHBOARD
➤ Query by one product category
➤ Query by one campaign (an ad set follows similar strategy)
Top Key Words Cloud Top Winning Brands Average Winning Bid
Top Impression Ads Top Click Ads Top CTR Ads
Current Budget Current spend Total Impression
Total Click Cost per Click Click/Impression
url：searchad.fun or http://107.23.229.58/

13. FURTHER CHALLENGE - ONLINE SYSTEM
Scaling up
➤ Sharding by campaignId
➤ Select top k in distributed way
➤ More replicas
A/B Testing
➤ Test different algorithms, give 10% load to server with
alternative algorithm

14. FURTHER CHALLENGE - REAL TIME MONITOR
➤ Log raw records to S3
➤ Other charge option(CPM) - optimize counting

15. FURTHER CHALLENGE
➤ Feed the real time click stream to update the prediction model
➤ Fast reaction to some special event
➤ More click, more income (advertiser charged by cost per click)

16. THANK YOU!

17. TECHNICAL INSIGHTS
➤ Increase search speed: Inverted index and features in cache
➤ Feature space design
➤ Joint event streams
➤ Design Cassandra schema for query efficiency
CampaignId(Partition), TimeStamp(Cluster), Chance, Impression, Click, AvgPrice

18. WHY CASSANDRA
➤ High write performance
➤ Easy scaling
➤ Flexible columnar schema
➤ Schema for query efficiency
CampaignId(Partition), TimeStamp(Cluster), Chance, Impression, Click, AvgPrice

19. WHY MEMCACHED
➤ Fast key value look up for online system
➤ Store - inverted index, rewrite query, CTR features

20. WHY SQL
➤ Clearly defined, discrete items with exact specifications
➤ Easy to do complex relational query
➤ Transactional workload

21. WHY SPARK STREAMING
➤ Micro batch, pre-defined time window
➤ Process events every 10 seconds

22. MODELS
➤ Word2Vector: Query Rewrite
➤ Logistic Regression: Predict pClick
➤ Gradient Boosting Tree: Predict pClick

23. ONLINE SEARCH AD PLATFORM
Search
Query
url：searchad.fun
Relevancy
Query
Understand
Filter
Ads
Select
Ads
Rank
Ads
Top K
Ads
Pricing
Allocate
Ads
P-click bid
Quality

24. Craft Demo
Grace Lu

25. OVERVIEW
➤ Problem
➤ Transform -> Join -> Reduce By State and Time
Customer information
<customer_id, name, street, city,state,zip>
123#AAA Inc#1 First Ave Mountain View CA#94040
Sales information
<timestamp,customer_id,sales_price>
1454313600#123#123456
State Sales
<State, Time, Total Sales>
State - hourly sales
State - daily sales
State - monthly sales
State - yearly sales

26. BASIC IDEA - TRANSFORM TO PAIR ADD
➤ Map Transformation - customer_rdd
➤ Map Transformation - sales_rdd
Customer information
<customer_id, name, street, city,state,zip>
123#AAA Inc#1 First Ave Mountain View CA#94040
(customer_id, state)
Sales information
<timestamp,customer_id,sales_price>
1454313600#123#123456
(customer_id, (year, month, date, hour, price))

27. BASIC IDEA - JOIN TWO RDD
➤ Join customer and sales on customer id
(customer_id, state)(customer_id, (year, month, date, hour, price))
(customer_id, (year, month, date, hour, price), state)

28. BASIC IDEA - REDUCE BY STATE AND TIME
➤ Extract
(customer_id, (year, month, date, hour, price), state)
Hourly: (state#year#month#date#hour, price))
Daily: (state#year#month#date##, price))
Monthly: (state#year#month###, price))
yearly: (state#year####, price))

29. BASIC IDEA - REDUCE BY STATE AND HOUR
➤ Reduce by state and hourly time to sum the sales
""" key: AL#2017#08#01#09 “""
hourly = joined_rdd.map(hourly_state_records).reduceByKey(lambda v1,v2)
pair: (customer_id, (year, month, date, hour, price), state) -> (state#year#month#date#hour, price)

30. BASIC IDEA - REDUCE BY STATE AND DAY/MONTH/YEAR
➤ Further aggregation - daily, monthly, yearly
""" key : AL#2017#08#01#”""
daily = hourly.map(lambda (k,v) : ('#'.join(k.split('#')[:4]) + '#', v))
.reduceByKey(lambda v1,v2: v1 + v2)
""" key : AL#2017#08##””"
monthly = daily.map(lambda (k,v) : ('#'.join(k.split('#')[:3]) + '##', v))
.reduceByKey(lambda v1,v2: v1 + v2)
""" key : AL#2017###”””
yearly = monthly.map(lambda (k,v) : ('#'.join(k.split('#')[:2]) + '###', v))
.reduceByKey(lambda v1,v2: v1 + v2)

31. OUTPUT ACTION - SAVEASTEXTFILE
➤ Union -> SaveAsTextFile
""" construct output file """
result = hourly.union(daily).union(monthly).union(yearly).map(lambda (k,v): k + '#' + str(v))
""" output format: state#year#month#day#hour#sales “""
result.saveAsTextFile(output_file)

32. FINAL LINEAGE
➤ RDD dependency
customer
daily
sales
hourlyjoined monthly yearly
union

33. REDUCE SHUFFLING - BROADCAST JOIN
➤ Broadcast HashMap - Map side join
(customer_id, state) -> collectAsMap -> broadcast map

34. (JOIN) IF BOTH DATASET CANNOT FIT INTO MEMORY
➤ Hard to avoid shuffling without prior knowledge
➤ Take advantage of pre-existing data distribution - like sorted
➤ Many missing values in inner join - pre-filter

35. REDUCE SHUFFLING - PARTITIONER
➤ Customer Patitioner
hourly
hourly
hourly
hourly
daily
daily
daily
daily
monthly
monthly
monthly
monthly
hourly
hourly
hourly
hourly

36. REDUCE SHUFFLING - PARTITIONER
➤ Data Locality
.reduceByKey(lambda v1,v2: v1 + v2, 4, state_partitioner)
hourly
hourly
hourly
hourly
daily
daily
daily
daily
monthly
monthly
monthly
monthly
hourly
hourly
hourly
hourly

37. REDUCE SHUFFLING - PERSIST
➤ Don’t do the re-computation every time
customer
daily
sales
hourlyjoined monthly yearly
union

38. THANK YOU
➤ FURTHER DISCUSSION