RealTime AdTech reporting & targeting with Apache Apex
•
5 likes•446 views
AdTech companies need to address data increase at breakneck speed along with customer demands of insights & analytical reports. At PubMatic we receive billions of events and several TBs of data per day from various geographic regions. This high volume data needs to be processed in realtime to derive actionable insights such as campaign decisions, audience targeting and also provide feedback loop to AdServer for making efficient ad serving decisions. In this talk we will share how we designed and implemented these scalable low latency realtime data processing solutions for our use cases using Apache Apex.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (14)
Similar to RealTime AdTech reporting & targeting with Apache Apex
Similar to RealTime AdTech reporting & targeting with Apache Apex (20)
Recently uploaded
Recently uploaded (20)
RealTime AdTech reporting & targeting with Apache Apex
- 1. Every ad. Every sales channel. Every screen. One platform. Ashish Tadose Senior Data Architect @ PubMatic Apache Apex committer RealTime AdTech Reporting & Targeting with Apache Apex
- 2. Agenda § About PubMatic § Reporting & Targeting use cases § Apache Apex overview § PubMatic’s Streaming use cases with Apache Apex § Comparing Apache Apex § Roadmap 2Proprietary and Confidential
- 4. About PubMatic ü PubMatic is a leading marketing automation software company for publishers. ü Through real-time analytics, yield management, and workflow automation, PubMatic enables publishers to make smarter inventory decisions and improve revenue performance.. ü Drives Innovation in AdTech 4Proprietary and Confidential
- 5. 5Proprietary and Confidential ad impressions served daily bids processed daily data processed daily data under management data center across geography 40B+ 350B+ 50TB 10PB6 Scale That Drives Results
- 6. General Analytics Dashboard Guaranteed Ad Server - Pacing engine AdServering - Impression capping report - Floor recommendation Inventory Discovery Machine Learning Reporting & Analytics Platform Ad – hoc Reports Audience Reports Brand Control Reporting Use cases
- 7. Lambda Architecture – Velocity & Volume 7 Data Data Sink Batch Eg : Hadoop Batch write , random read Real Time e.g. Storm Random read & write Query & Merge
- 8. Use cases for a real time solution ü Real-time reporting - Reporting of critical metrics around campaign monetization - Revenue, impression & click info - Aggregate counters & reporting on top N metrics - Low latency querying using Kafka in pub-sub model. ü Real-time Monitoring - Alerts on deal tracking & monetization - Campaign & deal health ü Real-time Learning - Using the lost bid insights for price recommendations. ü Allocation Engine - Feedback to ad serving for guaranteed delivery & line item pacing 8Proprietary and Confidential AdServer AdServer AdServer AdServer Kafka Cluster RealTime reporting data processing Processing for AdServer Feedback
- 20. Apache Apex @ PubMatic User Browser AdServer REST proxy REST proxy In prem AWS Real-time architecture Kafka Cluster Kafka Cluster Client logs Kafka Input (Auction logs) Kafka Input (Client logs) CDN (Caching of logs) ETL operator ETL operator Filter Operator Filter Operator Dimensions Aggregator Dimensions Aggregator Dimensions Store Query Query Result Kafka Cluster Auction Logs Client logs Middleware Auction Logs Client logs Kafka Messages Kafka Messages Decompress & Flatten Decompress & Flatten Filtered Events Filtered Events Aggregates Query from MW Query Query Results AdServer
- 23. Confidential & Proprietary Comparing with other Streaming platforms 23
- 26. Apex vs Spark Streaming vs Flink 26Proprietary and Confidential Apache Apex Spark Streaming Apache Flink Release Recently graduated from Apache incubation Spark Streaming Feb 2014 Spark major 1.0 release since July 2014 Graduated in 2015 Major 1.0 release in March 2016 Commercial support DataTorrent DataBricks Hortonworks Cloudera MapR dataArtisian Companies using http://apex.apache.org/powered- by-apex.html GE Capital One Silver Spring Networks PubMatic ThreatMetrix FacilitiesSupplies Royal Bank of Canada Infosys TechMahindra MammothData CloudWick Synerzip Trace3 LeadFerret Target https://cwiki.apache.org/conf luence/display/SPARK/Power ed+By+Spark Large set of companies are using Spark however only few of them are using it for Spark Streaming as below AsiaInfo Big Industries Baidu Faimdata Kelkoo Localytics Opentable https://flink.apache.org/pow eredby.html Alibaba.com Bouygues CapitalOne Ericsson King Otto group ResearchGate Zalando
- 27. Apex vs Spark Streaming vs Flink 27Proprietary and Confidential Apache Apex Spark Streaming Apache Flink Streaming model Native – event based stream processing Micro-batching Native - event driven stream processing API Declarative APIs Lower level compositional API Declarative - higher order functions System optimizes topology itself Declarative - higher order functions System optimizes topology itself Latency Very LOW HIGH Very LOW Throughput HIGH HIGH HIGH Query-able In-memory aggregate store In memory Dimension Store operator No native support Can be achieved through DataFrames - not efficient Framework Managed distributed in-memory store Aggregate store snapshotting HDHT – good No native support Can be achieved by savings DataFrames in parque format - not efficient StateBackend holds in-flight data in the TaskManager’s memory. Apache Community OK GOOD GOOD
- 28. Resources • PubMatic - https://pubmatic.com/ • PubMatic blog - http://pubmaticblog.com/ • Apache Apex - http://apex.apache.org/ • Subscribe to forums • Apex - http://apex.apache.org/community.html • DataTorrent - https://groups.google.com/forum/#!forum/dt-users 28Proprietary and Confidential