Epiphany: Connecting Millions of Events to Thirty Billion Data Points in Real-Time
•
1 like•812 views
This document describes Epiphany, Rocket Fuel's real-time attribution platform. It connects millions of events to 50 billion data points to attribute conversions across devices and algorithms. It uses HBase to lookup impressions in milliseconds. Data flows from actions keyed by user/impression/conversion days to HBase and Hive tables. It enables idempotent attribution across advertisers and algorithms at scale in real-time.
Recommended
Recommended
More Related Content
What's hot
What's hot (10)
Viewers also liked
Viewers also liked (20)
Similar to Epiphany: Connecting Millions of Events to Thirty Billion Data Points in Real-Time
Similar to Epiphany: Connecting Millions of Events to Thirty Billion Data Points in Real-Time (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
Epiphany: Connecting Millions of Events to Thirty Billion Data Points in Real-Time
- 1. Epiphany: Connecting Millions Of Events To 50 Billion Data Points In Real-time Anirban Banerjee abanerjee@rocketfuel.com Shahansad Kp skp@rocketfuel.com
- 2. 01 ONLINE ADVERTISING ECOSYSTEM In a nutshell
- 3. Advertisers Publishers Users ZZZX Exchanges Page Request 1 Ad Request 2 Bid Request 3 Bid Response 4 Bid Win 5 Ad served 6 Ad served 7 0 Set preferences Click & Visit Serve Impression Convert (e.g. buy a product) Observers
- 4. 02 Attribution Mapping effects to causes
- 5. How Was This "Conversion" Achieved? - Identify the effect of every single impression across every medium during the customer’s journey - Needed by modeling, reporting, analysts, customers.
- 6. Last Touch Attribution Whole credit to a single event
- 7. Multi Touch Attribution Across Multiple Devices Partial credits across impressions.
- 8. Multiple Algorithms Algorithm 1. Algorithm 2. Algorithm 3.
- 9. Attribution using Advertiser Data
- 11. Action by Impression day Action by Conversion day Rocket Fuel Attribution Previous Day Advertiser Data Current Day Rocket Fuel Conversion Data Reattribution Data Flow & Data Democracy Analysts Downstream ETL Rocket Fuel Impression History Data
- 13. (|Conversions| * |Impressions| * |Algorithms|) Impressions Tens of Billions Advertiser reports Thousands Conversions Hundreds of millions Algorithms Hundreds O
- 14. 04 Epiphany Rocket Fuel attribution platform
- 16. HBase backed object lookup of impression table Powered by Blackbird Collections Lookup in milliseconds
- 19. Skipping filter on column qualifiers
- 20. Point updates to hive Point updates to a intermediate HBase table Periodically pulled to Hive
- 21. Epiphany Tables Action keyed by Impression day Action keyed by Conversion day Action keyed by User Id HBase Table Hive Table INDEXAction by Conversion day Action by Impression day
- 22. Intermediate Table Data Flow Action keyed by User Id Records with deltaTimestamp based scan Action keyed by Impression day Action keyed by Conversion day Old state of records with delta Point reads Computed “changes”
- 23. Idempotency Reducer_1_attempt_1 Reducer_2_attempt_1 Reducer_1_attempt_2 Job Hbase Idempotency at record level is necessary for correctness
- 24. Hive Table Data Flow Hbase intermediate table Hive table Snapshot Scan with prefix filter Snapshot using HBase admin
- 27. Test releases with HBase snapshots Monitor health of HBase instance Use WAL (Write ahead log)
- 28. Generic solution at scale One ring to rule them all - Multiple attribution algorithms - Cross-device scenario - Advertiser attribution data Faster availability, faster experiments More accessible data - e.g. point-readable actions
- 29. Anirban Banerjee abanerjee@rocketfuel.com Shahansad Kp skp@rocketfuel.com [Major Contributors] Abhijit Pol Savin Goyal Zhan Yuan WE ARE HIRING!!!