Measuring 2.0 - How to handle 100K events/sec - Berlin Buzzwords 2019
•
0 likes•404 views
These are the slides of my Berlin Buzzwords 2019 presentation that I gave on 2018 06-17. This talk is essentially about the choices we made in designing and implementing our streaming system that can do behavioral analytics. Note that the mentioned Useragent analyzer that is used in this project is an opensources Apache 2.0 licensed project of mine.See https://yauaa.basjes.nl for more information about this tool.
Recommended
Recommended
More Related Content
What's hot
What's hot (13)
Similar to Measuring 2.0 - How to handle 100K events/sec - Berlin Buzzwords 2019
Similar to Measuring 2.0 - How to handle 100K events/sec - Berlin Buzzwords 2019 (20)
Recently uploaded
Recently uploaded (20)
Measuring 2.0 - How to handle 100K events/sec - Berlin Buzzwords 2019
- 1. Niels Basjes | Principal IT Architect How to handle 100K events/sec Measuring 2.0
- 2. • Context • Why Measuring 2.0 (M2) • Measuring • Cause and effect • End-to-end pipeline • State machines Agenda
- 3. TU-Delft Computer Science (MSc) Nyenrode Business School (MSc) Software Developer (USoft) Research Scientist (NLR) Infra Architect (NLR) WebAnalytics Architect (Moniforce) IT Architect/Inventor (Bol.com) Contributor for Apache Hadoop, Pig, HBase, Flink, Beam, Parquet, … Apache Avro Committer & PMC Niels Basjes nbasjes@bol.com @nielsbasjes https://github.com/nielsbasjes
- 6. 20.074.985
- 9. Measuring interaction • What are we showing (and why)? • How are our visitors responding? • Pages • Products/Offers • Add to cart • Purchase • Advertising • Inspiration
- 10. Use cases • Dashboards • Personalization • Site optimization • Fraud prevention • Data Science • …
- 11. Why M2? We already have Omniture/GA…
- 12. ~ 3K-4K pages/sec ~ 30 events/page ~ 100K events/sec Q2 2019: Per day • 109 measurements • 2TiB data More details
- 13. JavaScript data is … • Broken … fundamentally broken • Measure side effect • Missing & Duplicate orders • Blockable • Intelligent Tracking Protection, Adblockers, Spiders, Hackers, … • “Boxed” SaaS solutions. • Not enough ‘eVars’. Loading extra labels needed for reporting.
- 15. Measurements are… too old. • Available once every 24 hours. • So personalization is a ‘day behind’. Useless inspiration: I was interested in this YESTERDAY
- 16. Data relevance decay Age of the data Valueofthedata Days WeeksMinutes
- 17. Time is not always important Crowd pattern analysis of website usage Building a better website for future visitors: Batch Individual pattern analysis of website usage Supporting and advising the current visitor: Realtime
- 18. Batch processing
- 20. M2 Measuring 2.0
- 21. It’s really all about… • Measuring • Better • Processing • Faster • Applications • More relevant
- 22. Goals of “Measuring 2.0” • Measure everything of our website • All interactions (also AJAX) • All channels (also mobile, email, …) • All countries • All details • All visitors (also Googlebot) • More reliable data • Lowest possible load on the client • Lowest possible latency (< 1 second)
- 23. Goals of “Measuring 2.0” • Developer • Easy to build • Easy to validate • Test automation • Business • Always measure everything • Data is “independent” • New questions are allowed
- 24. Goals of “Measuring 2.0” • Privacy by design • General Data Protection Regulation (GDPR) • Algemene verordening gegevensbescherming (AVG) • No long term profiling • Security • Avoid storing “login” info. • Business • Do long term profiling
- 25. THE goal of “Measuring 2.0” Make the best possible interaction data stream.
- 27. Measure
- 28. Where to measure? • Measure where “it” happens. • “In” the responsible “frontend” service! • Webshop • App API • Basket Service • Order Service • … • Usually NOT in the browser
- 29. Measure pages • Serverside • What is in the page • Clientside (Javascript) • What part was viewed • Screen resolution
- 30. Measure orders • Listen to Order events! • with website/app sessionid. • The “Order confirmation” page. • Just a viewing of the order.
- 31. Record everything at the start • Measure what “really” happens. • Keep all relevant details • Product: ProductId, Product type, … • Offer: OfferId, ProductId, Price, Condition, SellerId, … • Later joining on productid/offerid is “impossible”. • Webshop caching • Data volume / Extra latency
- 32. Cause and effect
- 33. • All single event entities • No correlations • No logical ‘cause and effect’ This is easy
- 34. Our usecases • Banner optimization • Look / Search Next page better banner • A/B testing • Show feature Use Buy product • Search Suggestions • Search Find Choose Buy product • Attribution modeling • Show ad Click Buy product • …
- 35. Behavioral analytics • Cause and effect • Action: We show something • Reaction: To click or not to click
- 36. Event ordering matters • Click banner , Buy product • Banner WAS (possibly) part of reason to buy. • Buy product, Click banner • Banner WAS NOT part of reason to buy. “WAS” or “WAS NOT” is based on The ordering of the events
- 37. Finite State machine • Simple, low latency, pattern detection • Ordered events
- 38. Pushdown automaton • State machine • with a memory stack • Simple, low latency, pattern detection • Ordered events
- 39. Event ordering matters • A fast temperature change is dangerous • should alert IMMEDIATELY • Delta stays in bounds • Expect “Ordered” while (curr = newEvent()) { if (tooBig(curr, prev)) sendAlert(); prev = curr; } -40 -20 0 20 40 60 80 100 T1 T2 T3 T4 T5 T6 T7 T8 T9 Temperature Delta This is a simple pushdown automaton
- 40. Event ordering matters • A fast temperature change is dangerous • should alert IMMEDIATELY Ordering problems • Many false positives ! • Many false negatives ! -40 -20 0 20 40 60 80 100 T1 T5 T7 T4 T2 T8 T3 T6 T9 Temperature Delta ! ! ! ! !
- 41. Repairing event ordering • Is hard • Needless complexity • Takes time • Buffer for the maximum ‘out-of-orderness’ period. • Several minutes • We want really low latency 123 4 56 78 9 Sliding time based sort buffer
- 42. Exactly once please • At least once • Need data deduplication • Is hard • Large memory buffer • Idempotent output • Takes time
- 43. We need ordering guarantees per session
- 44. Achieving end-to-end ordering... 1. The measuring point 2. Measurement transport 3. Measurement processing
- 45. The measuring point • Single entity • single measuring instance • Multiple instances • Multiple output buffers • Race conditions • Ordering problems
- 46. The measuring point In IOT: • One temperature sensor • one recording device At bol.com • One visitor • Single webshop instance • Session routing is a MUST have!! • Not perfect! • Impact negligible • “View” measurements • Orders
- 47. Transport
- 48. Message transport • We need ordering per session: FIFO • “Queue” or “Partitioned Queue” • Session pinned to a specific partition https://en.wikipedia.org/wiki/Queue_(abstract_data_type) Partitioned Queue
- 49. Many “Queue” are not a Queue ! https://en.wikipedia.org/wiki/Java_Message_Service https://stackoverflow.com/questions/16300353/activemq-lifo-ordering
- 50. Many “Queue” are not a Queue ! https://cloud.google.com/pubsub/docs/ordering
- 51. High volume partitioned queues • Apache Kafka • https://kafka.apache.org/ • Production ready • Apache Pulsar • https://pulsar.apache.org/ • Connector for Flink very new. • Pravega • http://pravega.io/ • Not yet production ready • Amazon Kinesis • Sorry, wrong cloud • Microsoft Event Hubs • Sorry, wrong cloud Azure Event Hubs
- 52. Use Google PubSub If ordering does not matter at-least-once is Ok. High IO ‘Distributed Set’
- 53. If ordering matters and/or exactly-once is needed High IO ‘Partitioned Queue’ Use Apache Kafka
- 54. Processing
- 55. Measurement processing Requirements • Low latency • Exactly once • Ordering guarantees • A pushdown automaton per session • Keyed Stateful processing • Where the ‘key’ is the ‘session id’
- 56. Choosing a Processing toolkit Apache Beam • Low latency … except • Exactly once by deduplication • NO ordering guarantees • NO natural keyed stateful processing • “Dynamic” scaling • Abstract Java API • Runs on • DataFlow • Flink
- 57. Choosing a Processing toolkit Apache Flink • Low latency • Exactly once • Ordering guarantees • Keyed Stateful processing • “Fixed” scaling • Easy Java API • Runs on • Hadoop • Kubernetes
- 58. Changes happen
- 59. Applications change! • New business • New insights • New wishes • New scope • New … The records will • get new fields • have obsolete fields
- 60. Data producerData producerData producer Streaming applications Data producer Streaming Interface Data consumers Data consumersData consumers Data consumers The real payload is “byte array” Multiple Applications Rolling upgrades Canary releases Multiple Applications Rolling upgrades Canary releases
- 61. Kafka persists messages • A message is retained until the TTL expired. • So a topic will contain several message versions! • With different fields V1 V2 V3 V4
- 62. So we need something to • Serialize records into bytes • Data types • Nested records • Bidirectional Schema evolution
- 63. Apache Avro is what we use !
- 64. Apache Avro (IDL Schema) Code generation
- 65. Avro Message format • Single record into bytes encoding • Designed for evolving streaming applications • Need schema database: • Key = 64bit long • Value = String The json version of the schema
- 66. Producing from Flink into Kafka
- 67. Consume from Kafka into Flink
- 68. Consume from Kafka into Flink
- 69. AnonymizedPersonal Measuring 2.0Browser Measuring2.0 JavaScript HTML Measure endpoint Kafka Sessionizer Kafka Kafka YearsAnonymize Personalization, Campaigning Search Suggest, RECO, … Analytics (Web, Price, …) Security Fraud prevention WebShop Build HTML Measuring2.0 SORSOROrders/… VisitId GeoIP Useragent Files/BQ Files/BQ
- 70. Using the data
- 72. The search funnel Search Find (Product page) Choose (Add to cart) Buy
- 73. Deeper = more relevant (Search) Find Choose Buy
- 74. High level M2 Stateful analysis Relevant events Suggestions Event to scored suggestions Suggestion Delta Is this a valuable event? Pushdown Automaton 1. Input: “Harry Potter” PDP 2. Score: “Harry Potter” 25 3. Suggest: “H” “Harry Potter”: 25 “Ha” “Harry Potter”: 25 “Har” “Harry Potter”: 25 Ordering is important Ordering is NOT important
- 75. Search Suggestions Default no search Searched Found Chosen Initial Search for “X” To PDP Add to Cart Within PDP Anythingelse The PDP is really a set of pages about the product, offers, reviews, … = Send out event that something useful happened Searched for “X” PDP for “X” ATC for “X”
- 77. Join us 70
Editor's Notes
- Just storing the globalid/offerid and later joining is impossible due to the size of the datasets, the required speed and caching.
- https://blog.scottlogic.com/2018/04/17/comparing-big-data-messaging.html