This document discusses using Node.js for analytics aggregation to count 30 billion requests per day. It begins by introducing the speaker and his company, which does advertising and collects data. It then discusses why analytics are needed for quality assurance and making data-driven decisions. It explains what aggregation is through examples and buckets. It also summarizes three existing analytics solutions: StatsD, ELK Stack, and GOYOO Stack, which combines StatsD, Elasticsearch, and Kibana. It concludes with a demo of collecting HTTP request data.
1. USING NODEJS TO COUNT 30
BILLION REQUESTS PER DAY
#NINGJS2016
GUILLAUME VERBAL @DOWEIG
2. USING NODEJS TO COUNT 30 BILLION REQUESTS PER DAY
WHO WE ARE
â–¸ We are mainly an advertising company
â–¸ Doing many more !
▸ networks, data collection/analysis, web tech, app generation…
â–¸ 250 employees and counting, (100 tech people)
â–¸ Based in the center of the world (Beijing)
â–¸ I am:
â–¸ french, living in Beijing
â–¸ a software engineer (5 years) wannabe data scientist
â–¸ really into the blockchain
8. WE JUST PUSHED SOMETHING ONLINE
THE WORLD IS FULL OF QUESTIONS
â–¸ Is it crashing?
â–¸ Is it fast?
â–¸ Is our feature awesome yet?
â–¸ Are we famous yet?
â–¸ Are we rich yet?
9. WE JUST PUSHED SOMETHING ONLINE
OUR APP THROUGH THE NAKED EYE
34. WHAT IS AGGREGATION
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
LEARNING THROUGH EXAMPLE CLIENT
BROWSER
SERVER
RESPONSE
REQUEST
35. WHAT IS AGGREGATION
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
LEARNING THROUGH EXAMPLE CLIENT
BROWSER
SERVER
RESPONSE
REQUEST
36. WHAT IS AGGREGATION
AGGREGATION BUCKETS
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
37. WHAT IS AGGREGATION
AGGREGATION BUCKETS
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
SAFARI IE CHROME
38. WHAT IS AGGREGATION
AGGREGATION BUCKETS
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
SAFARI IE CHROME
39. WHAT IS AGGREGATION
AGGREGATION BUCKETS
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
SAFARI IE CHROME
1
40. WHAT IS AGGREGATION
AGGREGATION BUCKETS
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
SAFARI IE CHROME
1 2
41. WHAT IS AGGREGATION
AGGREGATION BUCKETS
id: 2,
browser: “IE”,
uri: “/login”,
timestamp: Date
id: 4,
browser: “IE”,
uri: “/”,
timestamp: Date
id: 1,
browser: “Safari”,
uri: “/”,
timestamp: Date
id: 3,
browser: “Chrome”,
uri: “/home”,
timestamp: Date
SAFARI IE CHROME
1 2 1
47. STATSD
BASIC CONCEPTS
â–¸ Our app will send some specialized logs to statsd
â–¸ Statsd will aggregate the logs and send them to the database regularly
52. ELK
BASIC CONCEPTS
â–¸ Our app will send every log to elastic search (using logstash)
â–¸ We will use kibana to send aggregation queries to elastic search.
â–¸ Kibana will make the aggregation response very visual