Tapad uses Hadoop to analyze large volumes of bid request data to predict available monthly impressions for new advertising campaigns. They tested several Hadoop technologies including Hive, Impala, Scalding, and Avro for serializing and compressing data. Hive was used initially for its SQL interface. Impala showed promise as the most developed interactive querying option. The company also considered Hortonworks Stinger and plans to evaluate it further. Hardware consisted of commodity servers with multiple CPUs, large memory, and RAID cards to store petabytes of bid request data across three data nodes and one master node.

1. HADOOP AT TAPAD
March 14, 2013
A Case Study
Mike Moss, VP Engineering
@michaelmoss

2. What is Tapad?
2
 Tapad is the first digital advertising solution for real-time mobile audience buying and multi-
screen targeting.
 Marketers use Tapad to obtain a unified view of their customers across smartphones,
tablets, computers and smart TVs, enabling more relevant and device-specific messaging.
 Tapad bridges devices together to create the Device Graph which enables Cross Platform
Targeting and Analytics

3. Device Graph Targeting Capabilities
 Retargeting
- Retarget PC visitors on mobile or tablet
 Location Targeting
- Geo-Fencing
- Airport Targeting
 Audience Targeting
- Economic (Income, Net Worth, Discretionary Income, Home Value, Charitable
Contributions, Invested Assets)
- Demographic (Age, Genders Present, Presence of Children, Ethnicity)
 Platform Targeting
- Platform (PC Web, Mobile Web, In-App, Connected TV)
- Device (Android, Android Tablet, Blackberry, Computer, Feature phones, iPad, iPhone,
Palm, Symbian, Windows Phone)
- Carrier (AT&T Wireless, MetroPCS, Sprint, T-Mobile, TracFone, Verizon Wireless, etc.)

4. Data at Tapad
• MySQL
• “CRUD” – Tapad UI and Campaign Manager
• Redis
• Counters – Revenue, Bid Requests, Impressions
• Aerospike
• Device Graph
• Vertica
• Impressions, Clicks, Aggregations - Reporting, ad-hoc queries

5. Use Case: Predict Available Monthly Impressions
for New Campaigns
 How can we predict how many monthly impressions a new advertiser can buy on our
platform?
D1 D2
D3
Advertiser
Home
Page
1 – Pixel for D1
2 - Device Graph Propagation
3 – Bid Request for D2
MonthlyUniquesNewAdvertiser
MonthlyUniquesSimilarAdvertiser
*MonthlyBid RequestsSimilarAdvertiser

6. Bid Requests
 At peak, we get over 150K bid requests/sec
 High Volume/”Low Value” data
 Complex data type (bid_sample_avro.json)
 Not sure of all the ways we would query it
 At a sampling rate of 1/1000, we are capturing 200MB/Hour
 …in other words: Perfect for Hadoop

7. Hadoop Ecosystem
 Hadoop Ecosystem – Heavily fragmented, lots of choices!
 Trends
- “Distro Wars” – Cloudera vs Hortonworks vs MapR
- Real-time, interactive ad-hoc querying – aka “Faster Hive”
- Apache Drill, Cloudera Impala, Stinger Initiative (YARN, Tez, ORCFile)
- Many influenced by Google Dremel paper
- All are similar and seek to improve on M/R expensive start-up time, avoid
shuffle/sort disk serialization where possible, as well as unnecessary M/R pipelines.
- New languages/frameworks
- Many more choices than just Pig and Cascading
- Scalding, Scoobi, Spark, Crunch/Scrunch
- Many influenced by Google Flume paper, seek to avoid awkwardness of the UDF
programming model, and experiment with richer typed data models (not just tuples)

8. Tapad Hadoop POC
 Some SQL, some code
 POC
- Hive
- Familiar SQL syntax
- Easy to get started
- Hue/Beeswax makes SQL on Hadoop easy to non-programmers
- Impala (Cloudera)
- Most developed of the pack (as of Feb 2013)
- Scalding (Twitter)
- “A Scala API for Cascading”
- Algebird
- Cloudera CDH4
 On our Radar
- Hortonworks – Stinger
- Scoobi
 Also tried
- Shark/Spark

9. Serialization
 Serialization Considerations:
- Parsing efficiency
- Schema evolution
- Compactness
- Complex type support
- Hadoop ecosystem support
 CSV
 JSON
 Avro – Like Protocol Buffers/Thrift, but better:
- Dynamic typing – No code gen required
- Untagged data – Since schema included with data, smaller serialization size
- No manually-assigned field IDs – Schema migrations are a breeze with presence of old
and new schemas

10. Compression
 Compression Considerations:
- Splittability
- Speed vs. Compression
- Hadoop ecosystem support
 gzip
 lzo
 Snappy
- “…aims for very high speeds and reasonable compression”
- Integrates seamlessly with Avro

11. Hive Demo
CREATE TABLE bids
ROW FORMAT
SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS
INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES ('avro.schema.literal'=‘<JSON SCHEMA HERE>’);
LOAD DATA LOCAL INPATH ‘bids.avro' INTO TABLE `bids`;

12. Impala Demo

13. Scalding
UnpackedAvroSource(args("input"), schema = None)
.read
.flatMapTo('request -> 'audienceId) { record: Tuple =>
val request: Tuple = record.getObject(0).asInstanceOf[Tuple]
val device: Option[Tuple] = Option(request.getObject(6).asInstanceOf[Tuple])
val audienceRecords: Option[ArrayList[Tuple]] = device.flatMap { record =>
Option(record.getObject(7).asInstanceOf[ArrayList[Tuple]])
}
audienceRecords.toSeq.flatMap { records =>
records.asScala.map(_.getString(0))
}
}
.groupBy('audienceId) { _.size('count) }
.groupAll { _.sortBy('count) }
.debug
.write(Tsv(args("output")))

14. Hardware
14
 1 Master Node – 1U
- 2 x Intel Xeon E5-2620 6-Core 2GHz
- 64GB DDR-1600 RAM
- LSI 9240-8i 8-Port RAID Card
- 2 x 1TB Seagate Constellation.2 SAS
 3 Data Nodes – 2U 12 HD Bays
- 2 x Intel Xeon E5-2620 6-Core 2GHz
- 64GB DDR-1600 RAM
- LSI 9207-8i 8-Port RAID Card
- OS Drive: 100GB Intel DC 3700
- Data Drives: 12 x 3TB Seagate Constellation CS SATA

15. References
15
Cloudera vs. Hortonworks: http://wikibon.org/wiki/v/The_Hadoop_Wars:_Cloudera_and_Hortonworks%E2%80%99_Death_Match_for_Mindshare
Dremel:
http://research.google.com/pubs/pub36632.html
http://www.quora.com/How-will-Googles-Dremel-change-future-Hadoop-releases
FlumeJava: http://faculty.neu.edu.cn/cc/zhangyf/cloud-bigdata/papers/big%20data%20programming/FlumeJava-pldi-2010.pdf
Hadoop Ecosystem (Mar 2013): http://gigaom.com/2013/03/05/the-hadoop-ecosystem-the-welcome-elephant-in-the-room-infographic/
Hardware:
http://hortonworks.com/blog/why-not-raid-0-its-about-time-and-snowflakes/
http://hortonworks.com/blog/best-practices-for-selecting-apache-hadoop-hardware/
Impala: https://ccp.cloudera.com/display/IMPALA10BETADOC/Impala+Frequently+Asked+Questions
Spark/Shark: http://www.cs.berkeley.edu/~matei/talks/2012/hadoop_summit_spark.pdf
Stinger: http://hortonworks.com/blog/100x-faster-hive/
SQL on Hadoop: http://gigaom.com/2013/02/21/sql-is-whats-next-for-hadoop-heres-whos-doing-it/
Tuples vs. Complex Types: http://www.quora.com/Apache-Hadoop/What-are-the-differences-between-Crunch-and-Cascading

16. Thank You
16
 Questions?
 Tapad is hiring!
- Data Scientists, Platform/Data/Frontend Engineers
- http://www.tapad.com/careers/
- michael.moss@tapad.com