This document provides a technical overview of the Graylog data processing engine in 9 sections. It describes the key components, including the transport layer which uses an append-only journal to prevent message loss, a high speed processing chain using ring buffers, a REST API that allows building custom front ends, an embedded Elasticsearch cluster for fast writing, and an index aliasing system to seamlessly switch between indexes. The goal is to provide the fastest machine data processing through an optimized architecture and attention to small performance details.

1. GRAYLOG ENGINEERING
DESIGN YOUR ARCHITECTURE.

2. README.
This is not a guide for the squeamish.
This is a peek for those who like to go off the beaten path, sometimes alone.
For those who aren’t afraid of pulling open the hood and getting their hands dirty.
This is the culmination of five years of engineering design in our hope to bring you
the fastest machine data processing engine on the planet.
Don’t call your sales rep, they won’t know the answers.
-GRAYLOG ENGINEERING

3. 1.
2.
3.
4.
4 1/2
5.
6.
7.
8. 9.
LEGEND:
1 & 2. LOG MESSAGES & LOAD
BALANCER.
3. TRANSPORT LAYER.
4. PROCESSING CHAIN.
4½ - REST API.
5. MONGODB REPLICA SET.
6. ELASTICSEARCH CLUSTER.
7. ANATOMY OF A SINGLE INDEX.
8. INDEX MODEL.
9. DEFLECTOR QUEUE.

4. 1 & 2, LOG MESSAGES & LOAD BALANCER.
tl;dr
We’re not going to spend any time here. Basically, send us any machine data
(structured or not) and use whatever load balancer you like.
The # of messages, their peak rates, average size and extractions performed will
affect performance, but we’ll cover that later.

5. 3, TRANSPORT LAYER.
This is the inputs and journal on top of the Graylog server. It consists of inputs from
the message cloud (this is our syslog stream, as well as other inputs). These get
pre-processed without user configurability into parts of a message.
While the journal is on disk (I/O), it is an *append only* journal where there is no
seek time. (Internally we re-use Apache Kafka code to do this - thanks LinkedIn).
The write “needle” is always close to the same point on the disk so it does not
constantly scan. This makes it blazing fast. You can turn it off, but we do not
recommend it.
Why we did this: Other systems do not have this, so they will lose messages coming
in when message spikes happen because the network layer will start to reject them
or your local memory will explode.

6. 4, PROCESSING CHAIN.
These messages are then taken and written into a process buffer, which is a ring
buffer. We are using the Disruptor library from LMAX, a high speed trading company
that relies on high speed and low latency.
Messages are then processed by the process buffer processor, where stream
routing and extracting of fields happens. This part can get CPU intensive! The filtered
message then goes into the output buffer (another ring buffer), then the output buffer
processor, and onwards to Elasticsearch (ES) or user defined output.
ProTip: Tuning the number of processors run per buffer is important and should
never exceed the number of CPU cores you have available for graylog-server.
Increase number of processors if you see too low throughput and try to focus on
process buffer processors because the output buffer usually does not need many. A
symptom of not enough processors is full buffers.

7. 4½ , REST API.
Why is this different than any other rest API?
This is the same API we use on our web front end, hence you can make any
read/write call we do in your own UI. Yup, you can build your own front end.
Also, it has to be high quality, because this is the same API we use ourselves day to
day. It is not like others where it is just an API that is provided for external users to
integrate with, built once and patched with duct tape every release. Not that we don’t
like duct tape….

8. 5, MONGO.
Then there is Mongo, which is storing only metadata: users, settings and
configuration data on all items: streams, dashboards, extractors, etc. Anything you
configure. If Mongo goes down, Graylog will continue to run. So, it is your choice
whether to include it in a high availability design.
Mongo recommends for HA scenario’s three instances of it. This is because if one
goes down then Mongo has to recommend a primary, and without two more it can
get confused between the first two. See Mongo Replication set for instructions.

9. 6, ELASTICSEARCH CLUSTER.
We connect to ES servers as an embedded ES node that does not store data. So,
we look and act like an ES node, and know about configuration data (indexes,
shards, etc) for each ES server.
When writing to ES and when you are not a node, you have encode and transmit
over the wire as HTTP and then JSON and then decode it, etc. As a node you can
send it in native format, and it is fast.
For HA, we recommend having at least one replica configured.

10. 7, ANATOMY OF AN INDEX.
A single index (In this example, Graylog Index #25), is broken into shards. This
means the index is broken up and the parts are run on different ES nodes. This
makes for faster searches because the query result can be computed on multiple ES
nodes in parallel.
An index can also have replicas configured. This means that each shard is mirrored
to other nodes, which is great for HA.

11. 8, INDEX MODEL.
Each index is numbered starting with 0 the first time. In a time series database, all
data is stored with a time stamp, and once it is stored it is not gone back to be re-
written (hence is marked as READ_ONLY vs WRITE_ACTIVE for performance). So,
messages are not gone back to be re-inserted. This makes it fast. Because of the
time based storage, this also means when you query it you must give a time bound
search (i.e. in the last hour…).
Pro Tip: So the size of these indexes matter when performance tuning. You don’t
want to make the indexes too big because then the searches will take much longer,
and you don’t want them too short for the same reason. The indexes should be sized
based on the amount of data a you have and how far you normally search.
Sometimes people use it for longer historical strategic type searches. It is important
to know and size this correctly.

12. 9, DEFLECTOR.
We write to an index alias called ‘deflector’ that can be atomically switched to a new
index. This allows us not to worry about having to stop message processing when
creating a new index because that is error-prone to manage (oh, index #25 is now
closed, ahh wait, okay the next one is #26, go ahead and write).
Why are we telling you this? Because, well, it’s these kinds of things that makes us
different. We are proud of thinking about all the small things that give you great
performance and stability, and hope you have enjoyed reading this as much as we
did writing it.