Download as PDF, PPTX
![Map
<?php
$document = "I'm a little teapot short and stout here is my handle here is my spout";
/**
* Outputs: [0,0,0,0,0,0,0,0,1,0,0,0,1,0,0]
*/
function map($target_word, $document) {
return array_map(
function ($word) use ($target_word) {
if ($word === $target_word) {
return 1;
}
return 0;
},
preg_split('/s+/', $document)
);
}
echo json_encode(map("is", $document)) . PHP_EOL;](https://image.slidesharecdn.com/metrics-150412193217-conversion-gate01/75/Drinking-from-the-Firehose-Real-time-Metrics-24-2048.jpg)
![Reduce
<?php
$data = [0,0,0,0,0,0,0,0,1,0,0,0,1,0,0];
/**
* Outputs: 2
*/
function reduce($data) {
return array_reduce(
$data,
function ($count, $value) {
return $count + $value;
}
);
}
echo reduce($data) . PHP_EOL;](https://image.slidesharecdn.com/metrics-150412193217-conversion-gate01/75/Drinking-from-the-Firehose-Real-time-Metrics-25-2048.jpg)
![while (buffer.length > 0) {
var char = buffer.shift();
if ('n' === char) {
queue.push(new Buffer(outbuf.join('')));
continue;
}
outbuf.push(char);
}
var i = 0;
var tBuf = buffer.slice();
while (i < buffer.length) {
var char = tBuf[i++];
if ('n' === char) {
queue.push(new Buffer(outbuf.join('')));
}
outbuf.push(char);
}](https://image.slidesharecdn.com/metrics-150412193217-conversion-gate01/75/Drinking-from-the-Firehose-Real-time-Metrics-46-2048.jpg)
![function plot(point) {
var points = svg.selectAll("circle")
.data([point], function(d) {
return d.id;
});
points.enter()
.append("circle")
.attr("cx", function (d) { return projection([parseInt(d.location.geopoint.lon), parseInt(d.location.geopoint.lat)])[0] })
.attr("cy", function (d) { return projection([parseInt(d.location.geopoint.lon), parseInt(d.location.geopoint.lat)])[1] })
.attr("r", function (d) { return 1; })
.style('fill', 'red')
.style('fill-opacity', 1)
.style('stroke', 'red')
.style('stroke-width', '0.5px')
.style('stroke-opacity', 1)
.transition()
.duration(10000)
.style('fill-opacity', 0)
.style('stroke-opacity', 0)
.attr('r', '32px').remove();
}
var buffer = [];
var socket = io();
socket.on('geopoint', function(point) {
if (point.location.geopoint) {
plot(point);
}
});](https://image.slidesharecdn.com/metrics-150412193217-conversion-gate01/75/Drinking-from-the-Firehose-Real-time-Metrics-72-2048.jpg)
Uploaded bySamantha Quiñones
Drinking from the Firehose - Real-time Metrics
The document discusses the development of real-time data metrics and content performance testing, focusing on methods like multivariate testing and state monitoring to optimize user engagement. It explores the management of large datasets using technologies such as Hadoop and RabbitMQ, highlighting the challenges and solutions in data processing and visualization. Additionally, it outlines findings from various pipeline versions regarding throughput and performance improvements in handling real-time data streams.
More Related Content
Similar to Drinking from the Firehose - Real-time Metrics
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
Drinking from the Firehose - Real-time Metrics
- 1. Drinking from theFirehose Real-Time Metrics Samantha Quiñones
- 2.
- 4. “How would youlet editors test how well different headlines perform for the same piece of content?”
- 5. Measuring User Behavior •Application path • Use patterns • Mouse & attention tracking
- 7. Multivariate Testing • Sortall users in to groups • 1 control group receives unaltered content • 1 or more groups receive altered content • Measure behavioral statistics (CTR, abandon rate, time on page, scroll depth) for each group
- 8. State Monitoring • Debugging •Load Monitoring
- 9.
- 10. • Augmented intelligencefor content creators • Quality prediction
- 11. What if contentcould change itself based on the weather?
- 12.
- 13. How big isbig?
- 14.
- 15.
- 16.
- 17.
- 18. 25 megabytes / second Ata rate up to
- 19.
- 20.
- 21. Hadoop • Framework fordistributed storage and processing of data • Designed to make managing very large datasets simple with… • Well-documented, open-source, common libraries • Optimizing for commodity hardware
- 22. Hadoop Distributed FileSystem • Modeled after Google File System • Stores logical files across multiple systems • Rack-aware • No read-write concurrency
- 23. MapReduce • Framework formassively parallel data processing tasks
- 24. Map <?php $document = "I'ma little teapot short and stout here is my handle here is my spout"; /** * Outputs: [0,0,0,0,0,0,0,0,1,0,0,0,1,0,0] */ function map($target_word, $document) { return array_map( function ($word) use ($target_word) { if ($word === $target_word) { return 1; } return 0; }, preg_split('/s+/', $document) ); } echo json_encode(map("is", $document)) . PHP_EOL;
- 25. Reduce <?php $data = [0,0,0,0,0,0,0,0,1,0,0,0,1,0,0]; /** *Outputs: 2 */ function reduce($data) { return array_reduce( $data, function ($count, $value) { return $count + $value; } ); } echo reduce($data) . PHP_EOL;
- 26. Hadoop Limitations • Hadoopjobs are batched and take significant time to run • Data may not be available for 1+ hours after collection
- 27. “How would youlet editors test how well different headlines perform for the same piece of content?”
- 28. Consider Shelf-life • Mostarticles are relevant for < 24 hours • Interest peaks < 3 hours
- 29.
- 30.
- 33. Version 1 (PoC) CollectorCollectorCollectorStreamerCollectorCollectorCollectorReceiver CollectorCollectorCollectorStatsD Cluster ElasticSearch
- 35. this.visit = function(record){ if (record.userAgent) { var parser = new UAParser(); parser.setUA(record.userAgent); var user_agent = parser.getResult(); return { user_agent: user_agent } } return {}; };
- 37. Findings • Max throughputper collector: 300 events/second • ~70 receivers needed for prod • StatsD key format creates data redundancy and reduced data richness
- 38. Version 1 (PoC) CollectorCollectorCollectorStreamerCollectorCollectorCollectorReceiver CollectorCollectorCollectorStatsD Cluster ElasticSearch
- 39. Transits & Terminals •Transits - Short-term, in-memory, volatile storage for data with a life-span up to a few seconds • Terminals - Destinations for data that either store, abandon, or transmit
- 40. An efficient real-timedata pathway consists of a network of transits and terminals, where no single node acts as both a transit and a terminal at the same time.
- 41. StatsD • Acts asa transit, taking data and passing it along… • BUT • Acts as a terminal, aggregating keys in memory and becoming a transit after a time or buffer threshold.
- 42. Version 2 CollectorCollectorCollectorStreamer CollectorCollectorCollectorReceiverElasticSearchRabbitMQ
- 44. RabbitMQ • Lightweight messagebroker • Allows complex message routing without application-level logic • Can buffer 90-120 seconds of traffic
- 45. Version 2 • Eliminatedeventing and improved performance • Replaced StatsD with RabbitMQ • Data records are kept together • No longer works with Kibana (sadface)
- 46. while (buffer.length >0) { var char = buffer.shift(); if ('n' === char) { queue.push(new Buffer(outbuf.join(''))); continue; } outbuf.push(char); } var i = 0; var tBuf = buffer.slice(); while (i < buffer.length) { var char = tBuf[i++]; if ('n' === char) { queue.push(new Buffer(outbuf.join(''))); } outbuf.push(char); }
- 47. Findings • Max throughputper collector: 600 events/second • ~35 receivers needed for prod • Micro-optimized code became increasingly brittle and hard to maintain as custom logic was needed for every edge case
- 48. Version 2 CollectorCollectorCollectorStreamer CollectorCollectorCollectorReceiverElasticSearchRabbitMQ
- 49. Need to GetSerious • Very high throughput • Multi-threaded worker pool with large memory buffers • Static & dynamic optimization • Efficient memory management for extremely volatile in-memory data • Eliminate any processing overhead. Receiver must be a Transit
- 50. And also… • NotGoLang (because no one on the team is familiar with it) • Not Rust (because no one on the team wants to be familiar with it) • Not C (because C)
- 51.
- 53. Why Java? • Solidstatic & dynamic analysis and optimizations in the S2BC & JIT compilers • Clients for the stuff I needed to talk to • Well-supported within AOL & within my team
- 54.
- 56. public class StreamReader{ private static final Logger logger = Logger.getLogger(StreamReader.class.getName()); private StreamerQueue queue = new StreamerQueue(); private StreamProcessor processor; private List<StreamReader.BeaconWorkerThread> workerThreads = new ArrayList(); private RtStreamerClient client; public StreamReader(String streamerURI, AmqpClient amqpClient, String appID, String tpcFltrs, String rfFltrs, String bt) { ArrayList queueList = new ArrayList(); this.processor = new StreamProcessor(amqpClient); byte numThreads = 8; for(int i = 0; i < numThreads; ++i) { StreamReader.BeaconWorkerThread worker = new StreamReader.BeaconWorkerThread(); this.workerThreads.add(worker); worker.start(); } queueList.add(this.queue); this.client = new RtStreamerClient(streamerURI, appID, tpcFltrs, rfFltrs, bt, queueList); } }
- 57. public class StreamProcessor{ private static final Logger logger = Logger.getLogger(StreamProcessor.class.getName()); private AmqpClient amqpClient; public StreamProcessor(AmqpClient amqpClient) { this.amqpClient = amqpClient; } public void send(String data) throws Exception { this.amqpClient.send(data.getBytes()); logger.debug("Sent event " + data + " to AMQP"); } }
- 58.
- 59. Findings • Max throughputper collector: 2600 events/second • ~10 receivers needed for prod
- 61. Why ElasticSearch • Open-sourceLucene search engine • Highly-distributed storage engine • Clusters nicely • Built-in aggregations like whoa
- 62. Aggregations • Geographic Boxing& Radius Grouping • Time-Series • Histograms • Min/Max/Avg Statistical Evaluation • MapReduce (coming soon!)
- 63. • How manyusers viewed my post on an android tablet in portrait mode within 10 miles of Denton, TX? • What is the average time from start of page-load to first click for readers on linux desktops between 3am and 5am? • Given two sets of link texts, which has the higher CTR for a randomized sample of readers on tablet devices?
- 64. Browser to Browserin < 5 seconds
- 65. But wait… Is that“real-time”?
- 67. Real-Time for Real •Live analysis of data as it is collected • Active visualization of very short-term trends in data
- 68. Potential Problems • Smallsample sizes for new datasets / small analysis windows • Data volumes too high for end-user comprehension • Data volumes too high for end-user hardware/network connections
- 69.
- 71. D3JS • Open-source datavisualization library written in JavaScript
- 72. function plot(point) { varpoints = svg.selectAll("circle") .data([point], function(d) { return d.id; }); points.enter() .append("circle") .attr("cx", function (d) { return projection([parseInt(d.location.geopoint.lon), parseInt(d.location.geopoint.lat)])[0] }) .attr("cy", function (d) { return projection([parseInt(d.location.geopoint.lon), parseInt(d.location.geopoint.lat)])[1] }) .attr("r", function (d) { return 1; }) .style('fill', 'red') .style('fill-opacity', 1) .style('stroke', 'red') .style('stroke-width', '0.5px') .style('stroke-opacity', 1) .transition() .duration(10000) .style('fill-opacity', 0) .style('stroke-opacity', 0) .attr('r', '32px').remove(); } var buffer = []; var socket = io(); socket.on('geopoint', function(point) { if (point.location.geopoint) { plot(point); } });
- 75. By the way… xn= x + (r * COS(2π * n / v)) yn = y + (r * COS(2π * n / v)) where n = ordinal of vertex and where v = number of vertices and x,y = center of the polygon
- 76. var views =0; var socket = io(); socket.on('pageview', function(point) { views++; }); function tick() { data.push(views); views = 0; path .attr("d", line) .attr("transform", null) .transition() .duration(500) .ease("linear") .attr("transform", "translate(" + x(0) + ",0)") .each("end", tick); data.shift(); } tick();
- 77.
- 79.
- 80. Receiver Layer Receiver Buffer/Transit Processing& Routing Layer Processing & Routing Transit Storage Engine End-User Consumable Queues Layers are • Geographically decoupled • Capable of independent scaling • Fully encapsulated with no cross- layer dependencies
- 81. Interfaces Input Stream (Java) Routing(node.js) Filtering (node.js) Aggregation (PHP) Visualization (D3JS) MV Testing (PHP)
- 82.
- 83. Where are weNow? • It took 6 months to build a rock-solid data pipeline • Entry points from: • User data collectors • Application code
- 84. That was theeasy part.
- 85.
- 86. • Live debugging& runtime profiling • Embeddable visualizations • On-demand stream filters • Predictive performance analysis • Real-time sentiment analysis
- 87.
- 88.