Wayfair currently uses both Graphite and InfluxDB as a time series platform - The data is used by their developers, business stakeholders, by their internal alerting engine. Most importantly their 24x7 Ops Monitoring Center is using this data to constantly analyze the vital signs of Wayfair’s IT infrastructure and storefront operations.

1. May 15, 2018
The Four R’s of Metrics Delivery
Jim Hagan

2. 2
A Little Background
• Wayfair currently uses both Graphite and InfluxDB as a time series
platform
• We send a very diverse set of event trackers, timers, and other system
metrics from over 2000 VMs running hundreds of applications.
• Our systems are spread across three major data centers. The data is
used by our developers, business stakeholders, by our internal alerting
engine.
• Most importantly our 24x7 Ops Monitoring Center is using this data to
constantly analyze the vital signs of Wayfair’s IT infrastructure and
storefront operations.

3. 3
A Little Background
Our legacy data pipeline was an elaborate series of services relaying data
from data center to data center over UDP (used to avoid blocking calls from
the client). This configuration works very fast (and even supported
replication) but lacks a number of the elements we are looking for in our
metrics pipeline.
Our next generation pipeline takes advantage of Kafka and the Telegraf
streaming service to create a more robust data topology. Essentially this
allows us to explicitly implement...

4. 4
A Little Background
The Four R’s:
• Resiliency
• Redundancy
• Routability
• Retention

5. 5
The Four R’s
How we want to think of our our data pipeline...
Application(s)
Time Series
Database
Sweet Insights!

6. 6
The Four R’s
How it really is (current generation)...

7. 7
The Four R’s
(previous generation)...

8. 8
The Four R’s
What requirements are driving that architecture…
• Receive data from anywhere in simple text format (dynamic schemas) with a low
overhead network protocol.
• Handle in the range of 100 million data points per second (in the aggregate)
• Blacklist, whitelist and re-direct into more than one stream to achieve scaling and
other business objectives
• Handle spikes of 3x to 5x average peak traffic
• Buffer that raw data for a reasonable amount of time (say 24 hours)
• Ingest data when and where we please
• Make 99% of data available for time series queries within 30 seconds.

9. 9
The Four R’s
Breaking those requirements down conceptually we arrive at the “four R’s”
• Routability (keep, drop, redirect)
• Retention (keep data in pre-digested, or final format for some time)
• Resilience (survive network or DC failure, recover data after a DB failure,
survive massive flood of data)
• Redundancy (replication of raw data for failover and purpose built dbs)

10. 10
The Four R’s
Conceptual architecture to support the four R’s
LOAD
BALANCE
RECEIVE/
HANDLE
BUFFER INGEST PERSIST
Resilience
Routability
Redundancy
Retention Redundancy
Resilience
Routability Retention
Resilience Resilience Resilience
Routability Routability

11. 11
The Four R’s
Conceptual architecture to support the four R’s
LOAD
BALANCE
RECEIVE/
HANDLE
BUFFER INGEST PERSIST
NGINX
(UDP LB)
>> UDP
Telegraf
Receiver
>> UDP
Kafka Local
>> TCP
Telegraf Ingest
>> TCP
InfluxDB
>>TCP
Kafka
Aggregate
>> TCP

12. 12
The Four R’s
NGINX
(UDP LB)
Telegraf
Receiver
(>> UDP,
<< TCP)
Kafka Local
>> TCP
Telegraf Ingest
>> TCP
InfluxDB
>>TCP
Kafka
Aggregate
>> TCP
We use the UDP load balancing plugin for Nginx. This allows us to take very high data rates including 3 to 5
x spikes we referred to earlier and efficiently route them to an array of telegraf hosts. 3 udp load balancers
can feed into dozens of telegraf hosts.
In addition we can use different port designations to route traffic, so for example, let’s say we took all UDP
traffic into port 8094 and sent it to one array of receivers and all that into 8095 into another array of
receivers.
This is giving us RESILIENCY and a means of top level ROUTABILITY.

13. 13
The Four R’s
NGINX
(UDP LB)
Telegraf
Receiver
(>> UDP,
<< TCP)
Kafka Local
>> TCP
Telegraf Ingest
>> TCP
InfluxDB
>>TCP
Kafka
Aggregate
>> TCP
We use several features of telegraf to perform basic traffic shaping…
We use TAG and measurement filters to
1. drop certain metrics
2. keep certain metrics
3. route certain metrics to a specific Kafka topic
4. route to specific Kafka brokers
We are getting both RESILIENCE and ROUTING in this layer.

14. 14
The Four R’s
NGINX
(UDP LB)
Telegraf
Receiver
(>> UDP,
<< TCP)
Kafka Local
>> TCP
Telegraf Ingest
>> TCP
InfluxDB
>>TCP
Kafka
Aggregate
>> TCP
We use several features of telegraf to perform basic traffic shaping…
We use TAG and measurement filters to
1. drop certain metrics
2. keep certain metrics
3. route certain metrics to a specific Kafka topic
4. route to specific Kafka brokers
We are getting both RESILIENCE and ROUTING in this layer.

15. 15
The Four R’s
NGINX
(UDP LB)
Telegraf
Receiver
(>> UDP,
<< TCP)
Kafka Local
>> TCP
Telegraf Ingest
>> TCP
InfluxDB
>>TCP
Kafka
Aggregate
>> TCP
The local Kafka layer gives us an immediate place to store the metrics coming into the system. No
expensive processing needs to be applied to the data yet. We configure several different “mirroring”
services to copy different topics from local Kafka instances to what we call “Aggregate” kafka instances. All
of this communication is happening with minimal transformation.
We are getting both RESILIENCE and ROUTING and RETENTION in this layer.

16. 16
The Four R’s: Kafka Local/Aggregate Mirroring Topology
C1 Kafka
(write-only)
C1 Kafka
(write-only)
C1 Kafka
(write-only)
C3 Kafka
(read-only)
C3 Kafka
(read-only)
C3 Kafka
(read-only)
Seattle Boston Europe The C1 Clusters are used for write-only in the
local data centers. I addition each data center
has an “aggregate” cluster (C3) for consumption
of the integrated data stream. So if I’m in Boston
our Telegraf ingest layer can consume data from
all three data centers.
LOCAL
AGGREGATE

17. 17
The Four R’s
NGINX
(UDP LB)
Telegraf
Receiver
(>> UDP,
<< TCP)
Kafka Local
>> TCP
Telegraf Ingest
>> TCP
InfluxDB
>>TCP
Kafka
Aggregate
>> TCP
We have a second layer of Telegraf acting as a Kafka consumer. This allows us to subscribe to topics that
we care about and route them to the DB of our choice. We can also deploy multiple layers of ingest and
populate multiple databases.
We are getting both RESILIENCE and ROUTING and REDUNDANCY in this layer.

18. 18
The Four R’s
Some architectural recipes (Dynamic Topic Routing)
NGINX
(UDP LB)
Kafka Topic A
(receives metric A)
Telegraf Ingest
Consume
Topic A
InfluxDB for
Topic A
Kafka Topic B
(receives metric B) Telegraf Ingest
Consume
Topic B
InfluxDB for
Topic B
Telegraf Receiver
Keep Metric A
and Metric B
Discard Metric C

19. 19
The Four R’s
Some architectural recipes (Redundancy)
InfluxDB for
Data Science
(Longer
Retention)
InfluxDB for
Alerts (Short
Retention)
Telegraf Ingest
(Topic A)
Telegraf Ingest
(Topic A)
Kafka

20. 20
Monitoring It All (Load Balancer Layer)

21. 21
Monitoring It All (Telegraf Layers)

22. 22
Monitoring It All (Telegraf Layers)

23. 23
Monitoring It All (Kafka)

24. 24
A Little Background
References
https://tech.wayfair.com/2018/04/time-series-data-at-wayfair/
https://docs.influxdata.com/telegraf/v1.6/
https://kafka.apache.org/