Loggly utilizes Apache Kafka for efficient log management, achieving high fault tolerance and real-time data processing on a massive scale with over 750 billion events logged. The technology enables log collection during network outages and provides flexibility through dynamically created queues, optimizing resource utilization and managing diverse customer logging behaviors. Ultimately, Kafka's deployment allows Loggly to focus on enhancing their service as the leading cloud-based log management solution.

1. Why Loggly Loves Apache
Kafka, and How We Use Its
Unbreakable Messaging for
Better Apache Log Storm
Management
Infrastructure Engineering Team
June 2014
| Log management as a service Simplify Log Management

2. What Loggly Does
World’s most popular cloud-based
log management service
§ More than 5,000 customers
§ Near real-time indexing of events
Distributed architecture, built on AWS
Initial production services in 2011
§ Loggly Generation 2 released in Sept 2013
| Log management as a service Simplify Log Management

3. Loggly: Addressing the first big data
problem every company faces
§ Centralized logging
and archival
§ Real-time processing,
analysis and
visualization
§ Monitoring, alerting
and troubleshooting
| Log management as a service Simplify Log Management

4. Agenda for this Presentation
§ The challenges of Log
Management at scale
§ Overview of Loggly’s
processing pipeline
§ Alternative technologies
considered
§ Why we love Apache Kafka
§ How Kafka has added
flexibility to our pipeline
| Log management as a service Simplify Log Management

5. The Challenges of Log Management at Scale
§ Big data
– >750 billion events logged to
date
– Sustained bursts of 100,000+
events per second
– Data space measured in
petabytes
§ Need for high fault tolerance
§ Near real-time indexing
requirements
§ Time-series index
management
| Log management as a service Simplify Log Management

6. Log Management Processing Pipeline:
Overview
Load Balancing
Kafka
Stage
2
Loggly
Custom
Module
| Log management as a service Simplify Log Management

7. Collectors Can Easily Outpace
Downstream Processes
Load Balancing
Kafka
Stage
2
Loggly
Custom
Module
§ Written in C++
§ Designed to ingest
massive data volumes
§ Need to collect
regardless of what’s
happening
downstream
| Log management as a service Simplify Log Management

8. Solution:
Queue That’s External to Collector
Load Balancing
Kafka
Stage
2
Loggly
Custom
Module
§ Based on Apache
Kafka
§ Highly performant
and reliable
| Log management as a service Simplify Log Management

9. Alternate/ Supplementary
Approaches Considered
§ Internal buffering in collectors
– Added complexity
§ Cassandra
– Not as good a queue as Kafka
§ Apache Storm
– In initial Gen2 architecture, removed after launch
| Log management as a service Simplify Log Management

10. The Secret to Log Management at Scale:
Keep It Simple, Stupid
Results:
§ Can process sustained rates of
100,000+ events per second per cluster
§ Average message 300 bytes
| Log management as a service Simplify Log Management

11. Why We Love
Kafka
| Log management as a service Simplify Log Management

12. What Attracted Us in the First Place
No single point
of failure
• Terabytes of data move through our Kafka cluster
every day without losing a single event
• We use age-based retention to purge old data on disks
Low latency • 99.99999% of the time our data is coming from disk
cache and RAM; only very rarely do we hit disk
Performance • Crazy good!
• We currently have a bunch of Kafka brokers running
on m2.xlarge instances backed by provisioned IOPS.
• One of consumer group (eight threads) which maps a
log to a customer can process about 200,000 events
per second draining from 192 partitions spread across
three brokers
Scalability • Ability to increase partition count per topic and
downstream consumer threads provides flexibility to
increase throughput when desired
| Log management as a service Simplify Log Management

13. How Our Kafka Crush Has Deepened
Distributed log
collection
• Local pods and collectors spread all over the Internet with
local Kafka deployments to collect data from customers
located all over world
• Can collect logs even when we lose connectivity
• When network comes back, Kafka sends the logs
downstream to the rest of the pipeline
More efficient,
effective
DevOps
• Deploying Kafka throughout pipeline makes it easy to
disable certain parts of system (for troubleshooting or
upgrades)
• No worrying that we will lose customer data
• Example: Add support for new log type into our
automatic parsing capabilities by turning off existing
parser, deploying new one, and processing logs that
Kafka has queued up
Controlling
resource
utilization
• Keep collectors as simple as possible for resilience and
reliability reasons
• Add intelligence into our pipelines using Kafka
| Log management as a service Simplify Log Management

14. Resource Utilization Example:
“Noisy Neighbors”
| Log management as a service Simplify Log Management

15. “Noisy Neighbors” are
Inherent to SaaS
§ Sending many times their “normal” level of
logging volume, inadvertently or because their
application is in big trouble
§ Routing logs to separate queue minimizes
impact on other customers
| Log management as a service Simplify Log Management

16. Kafka Queues Add Flexibility to Loggly
Pipeline
§ Because Kafka topics are very cheap from a
performance and overhead standpoint, we
can create as many queues as we want
§ Scaled to the performance we want
§ Optimizing resource utilization across the system
§ Because they can be created dynamically, we
can make business rules very flexible
§ Makes us confident that pipeline will scale as
customer data volumes do
| Log management as a service Simplify Log Management

17. Conclusion:
Kafka Frees Our Development Team
to Build Differentiating Features
§ Kafka deployment working without us thinking
about it
§ Plenty of other things to do to keep our
position as the world’s most popular cloud-based
log management service!
| Log management as a service Simplify Log Management

18. Does Log Management
Sound Hard? It Should!
Let us do the heavy lifting for you!
Try Loggly FREE for 30 days
About Us:
Loggly is the world’s most popular cloud-based log management solution, used by
more than 5,000 happy customers to effortlessly spot problems in real-time, easily
pinpoint root causes and resolve issues faster to ensure application success.
Visit us at loggly.com or follow @loggly on Twitter.
| Log management as a service Simplify Log Management

19. Did you like this presentation?
Head over to our blog for
more great content!
Take me to the Loggly Blog
| Log management as a service Simplify Log Management