Achieving High Load in Advertising Technology
•
4 likes•1,113 views
High Load consists of three factors: * Latency - The Speed of an individual request, business transaction or event * Throughput - The Scale required to process a number business transactions per time period a.k.a TPS * Availability - The system 99.XXX% availability 24/7/365 AdTech is the technology that supports Digital Marketing. The “free” internet services such as, Email, Facebook, Skype, etc. are funded by Digital marketing. In this presentation covers the technologies used and how they are implemented to see the High Load demanded by Digital marketing.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (18)
Similar to Achieving High Load in Advertising Technology
Similar to Achieving High Load in Advertising Technology (20)
More from Peter Milne
Recently uploaded
Recently uploaded (20)
Achieving High Load in Advertising Technology
- 7. AVAILABILITY is ALWAYS ON Always on Telephone Electrical power Water supply Sewer …or for the young
- 9. WHAT IS RTB 1. Publisher sells space called an Impression 2. Exchange conducts and Auction 3. Advertisers bid to fill this space
- 10. RTB - AUCTION Sell Side (supply) Buy Side (demand) OpenRTB Server Auction Service Sell Side Platform (SSP) Demand Side Platform (DSP) OpenRTB Client Bidder Service Advertiser Ad Server Agency, Trade Desk or In-house team Publisher Ad Server Publisher Ad Network (working for a publisher) Real Time Bidding (RTB) is the kernel of programmatic advertising
- 13. DSP TECHNOLOGY - COOKIE PROFILE STORE Secret weapon: Aerospike 800-900k/s reads/s, 100k/s writes < 1ms latency to read profile 8 billion cookies in 8 servers HA with redundancy Moved from Cassandra 3 years ago – Why? Easy to use Value for money 150 servers -> 8
- 16. WHAT IS A DMP The ways how we attribute profiles are following: Pixels on web pages Cookies Server to server Custom rules specified by data providers Cross-device algorithms Look-a-like algorithms A solution to store and attribute anonymous user profiles - 1st party and 3rd party Profiles are analyzed and grouped into Audiences and Segments (sets). Segments, and their cookies, used for Real Time Bidding.
- 17. DMP SIZE 4b cookies – 50% map to external providers (Google, AppNexus, MediaMath, etc) 60% of cookies have 1st party data (browser, language, country etc) 130b cookie-segment pairs relations between profiles by their attributes 70k of possible cookie attributes about 1k form data providers Cookie profile updates 50k/s 100k/s spikes 2x growth every year
- 19. DMP TECHNOLOGY Polyglot environment .NET – for API and some foreground processing Java/Scala – everywhere where we need scale Java Script/React– UI Spark – Parallel Processing Polyglot persistence Aerospike – for fast read and write of user profiles MongoDB – for metadata and fast UI Vertica – Analytics database MSSQL – for places where we need price accuracy Ceph (distributed storage) – for files HDFS – “Raw” data files Parquet – Efficient file format Redis – for statistics and probabilistic counters Kafka/Confluent – for data streams Wide number of technologies: “right tool for the right job” … maybe too many
- 20. SIZE PROBLEM - SOLUTION The main problem: How to process huge and constantly growing amount of data without downtimes. Technology Solution: Horizontally scalable, HA and Multi-datacenter Aerospike as real-time cookie store HDFS + Spark. Programmatic solution - analytics approaches as: Sampling Probabilistic algorithms Monitor as much as possible Graphite is our best friend to see the trends, anomalies, performance problems and predict potential bottlenecks in advance.
- 21. DevOps
- 22. SIZE Infrastructure 1300 bare metal servers across 6 DCs (mostly HPE ProLiant DL360s and DL380s) Private Cloud 2400 OpenStack VMs 1050 Hyper-V VMs OS Distribution Physical servers: 50/50 Windows/Linux (mainly CentOS) VMs: 80% Linux / 20% Windows
- 23. DATA CENTERS - GEO Bare metal 1 – Vilnius 3 – Copenhagen 1 – New York 1 – Hong Kong AWS São Paulo
- 25. PROVISIONING Physical servers: Foreman up to the OS, then Puppet takes over. VMs: Puppet is the officially supported method. Terraform to launch VMs. OS images are standardized puppet - same configuration for every server (NTP, firewall settings, etc).
- 27. DATA STORAGE
- 29. HIVE VS SPARK 0 10 20 30 40 50 60 70 80 90 100 110 15 minutes 1 hour 15 minutes 4 hours 1 day Time to aggregate one day in minutes Aggregation chunk size ONE DAY AGGREGATION Hive Spark 1.5 Spark 2.0 Hive Spark 30x faster
- 30. VERTICA Scalable Column Store - Great for complex multi-filter queries Total data volume ~250TB Average query latency 0.541 sec Biggest tables 100B rows 30-40k queries per hour Rack aware for HA Datacenter aware for even more Horizontally scalable 30% growth per year ~4 nodes to cluster SELECT avg(price) FROM tickstore WHERE symbol = ‘AAPL” AND date = ‘5/06/09’ Before Vertica We used to use MS BI stack- MS SQL+MS OLAP. “It was a nightmare to develop and support the solution, especially scaling out” Now zero complaints 5/05/09 5/06/09 5/05/09 5/06/09 143.74 143.75 37.03 37.13 Column Store - Reads 3 columns NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS NYSE NYSE NYSE NQDS AAPL AAPL BBY BBY
- 31. AEROSPIKE Row store – Great for primary key queries Total data volume ~15TB Average query latency - 0.9 millisecond Biggest set - 8B rows ~1 million queries per second Rack aware for HA and DC redundancy Horizontally scalable 55% growth each ~4 nodes to cluster per year Before Aerospike Constantly fighting with Cassandra Java GC problems Rebalancing problems Defrag/compaction problems SELECT device, geo, segments FROM cookie_store WHERE cookieId = 826701826
- 34. DATA FLOW Data is the blood Use Kafka & Confluent Kafka is the circulatory system Confluent is a healthy lifestyle
- 36. DATA STORAGE AND PROCESSING SAVE YOURSELF SOME PAIN Use Vertica as your column store Þ Scale, HA - complex queries Replace Hive with Spark Þ At least 20x faster on same hardware Parquet and HDFS as your raw data store Þ 30% less space Use Aerospike as your row store Þ Speed, Scale, HA - primary key
- 37. TECHNOLOGY TO WATCH Persistent RAM Graphics Processing Units (GPUs) Deep learningColumnar databases
- 40. Use Continuous Integration & Delivery We use: TeamCity & Rundeck Build in blue/green deployment & Feature toggling Develop a DevOps culture – think “how will it run in production” Hire good people – not entitled princesses Use an Agile process - Agile is: A frame of mind A flexible technique to manage a project Allows new information to alter outcome PROCESS
- 42. MURPHY’S LAW Anything you haven’t tested will break Entropy – 3rd law of thermodynamics
- 46. @AdformInsider