Download as PDF, PPTX
Uploaded byIdo Shilon
BDX 2016 - Tal sliwowicz @ taboola
The document discusses Taboola's data infrastructure and how it has evolved over time. Some key points: - Taboola processes large amounts of data (5+ TB daily) from 750M monthly users to power recommendations. - Earlier systems struggled to handle the scale and lacked analytics capabilities. The current system uses Apache Spark and Cassandra for real-time and historical data processing and analytics. - Data flows from servers into Apache Spark for joining and analysis, then into Cassandra for querying and Hadoop for long-term storage. This allows real-time and historical access and analytics across petabytes of user behavior data.
BDX 2016 - Tal sliwowicz @ taboola
- 1.
- 2.
- 3. You’ve Seen UsBefore! Enabling people to discover information at that moment when they’re likely to engage
- 4. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. Entertainment | Lifestyle Tech OurClients are All Around the Globe
- 5. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. 750M monthly unique users 100K+ Requests/sec 10B+ recommendation s/day 5TB+ Daily data REACH PROPERTY 95.5% Google Ad Network 87.8% Taboola 86.2% Google Sites 61.5% Facebook 60.3% Yahoo Sites 56.6% Outbrain 52% mobile traffic 48% desktop traffic US desktop users reached, 12/2015 Taboola in Numbers
- 6. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. Context Metadata Region-based Location Recommendations User Behavior CookieData Collaborative Filtering Bucketed Consumption Groups CONTENT RECOMMENDATION ENGINE Social Facebook / Twitter API The Recommenda4on Engine
- 7. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. Taboola’s Discovery Platform TrafficAcquisition Business Dev.! Sponsored Content Editorial! Newsroom Sales! Native Ads Audience Dev. Product! Personalization Data & Insights!
- 8. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. • Events and logs (rawdata) wriPen directly to DB • Recs Are read from DB • Crashed when CNN launched Taboola 2007 Frontend FE Server
- 9. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. • Same as before, but without direct write to DB • Switching to bulk load • But – Very Basic Repor4ng, not scalable Taboola 2007.5 Frontend Bulk Load FE Server
- 10. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. • Introduced a semi real4me events parsing services: Session Parser and Session Analyzer • Divided analysis work by unit (session) • Files were pushed from RecServer(s) to Backend processing • Files are gzip textual INSERT statements • But – not real 4me enough Taboola 2008 Frontend NFS Backend FE Server SessionParser SessionAnalyzer Write Summarized Data Write rawdata Read session files Read rawdata Write session files
- 11. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. • Made a leap towards real-4me stream processing • Unified Session Parser and Session Analyzer to an in- memory service (without going through disk) • Made drama4c op4miza4on to memory alloca4on and data models • Failure safe architecture - can endure data delays, front-end servers’ malfunc4on • No direct DB access - key for performance, only using bulk loading for loading hourly data Taboola 2010 Frontend NFS Backend FE Server Session Parser + Analyzer Write Hourly Data (Bulk Loading) Write rawdata Read rawdata
- 12. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. • Mul4 DC • Roughly same architecture • Increasing backend growth by scaling in (monster machines) • Introduced real-4me analyzers • Introduced sharding • Moved to lsync based file sync • Introduced Top Reports capabili4es Taboola 2011-2013 Frontend Lsync Backend FE Server Session Parser + Analyzer Write Hourly Data (Bulk Loading) Write rawdata Read rawdata
- 13.
- 14. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. • Lots of incoming traffic (100K requests/sec) • Data (5+ TB / day): • Personalized served recommenda4ons – per user, per page view • Events - What the user actually read and what he did • The data needs to be joined and processed in real 4me • Campaigns Management • Recommenda4ons • Billing • Reports • Etc. • The data needs to be available for offline research Our Data Requirements
- 15.
- 16. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. • We care about sessions - chain of page views and events for a specific user • Length can be hours or even days • We care about users – chain of sessions across sites • Length can be days or even months • Stateless Applica4on – single user data is sent from mul4ple data centers and mul4ple servers • No determinis4c affinity to a server or DC • Order isn’t guaranteed • Must be robust and automa4cally deal with late arrivals • “Exactly once” seman4cs Challenges
- 17.
- 18.
- 19. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. • Par44on key - session start hour + user bucket (0-9,999) • Clustering key - publisher_id, user_id, session_id, view_id, data_type, data_hash • Data Type - MULTI_REQUEST, USER_EVENT, ACTION_CONVERSION, … • Data - blobs of protobuff • Results: • All the data of a single session is in one place, regardless of 4me of arrival • Idempotent process - if same message is received twice it overruns the previous arrivals due to same hash id • Sampling is built-in to the model Table Model in C*
- 20.
- 21. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. • Raw data – real 4me full access to the raw data, not just aggregated data • Week of data (~35TB) - 2 hours to analyze and report • 10 physical nodes , 320 Cores, 2.5TB memory, SSDs • Analyzing 1% sample of the users reduces this linearly (par44on key) • Analyzing a single publisher which is 1% of the data reduces this almost linearly (clustering key) • Repor4ng – minutes for availability of full repor4ng vs. hours • Suppor4ng our growth – Spark as a distributed compu4ng engine is very strong, easy to scale and extend Before vs. Ayer
- 22. Copyright©2016 The Nielsen Company. Confiden4al and Proprietary. • Long term data access – Hadoop, Cassandra and BigQuery provide a solu4on we did not have before • Analy4cs engine – the move from MySQL to Ver4ca (as an MPP engine) allows us to support complex queries over very large data sets • Algorithmic Research and Modeling – we are now capable of in depth analysis on mul4ple dimensions across long 4me periods Before vs. Ayer - Cont.
- 23.