Peter Czanik: syslog-ng - from log collection to processing and infomation extraction
LOADays 2015.
After a short introduction to system logging, we will show how the current log messages look like, and what the problem is with this free text format. Next, we will introduce you the powerful concept of name-value pairs, and how you can extract useful information from your logs by parsing log messages into name-value pairs. Next we will demonstrate the flexibility of syslog-ng’s message parsers (patterndb, csv and JSON parsers), and show you how to create patterns using a text editor or a GUI. This can also be used to overwrite sensitive information due to privacy regulations. At the end, you will learn about the Perl/Python/Lua/Java bindings of syslog-ng Open Source Edition, how value pairs can be passed to them, and some reference applications written for syslog-ng.
SCaLE 2016 - syslog-ng: From Raw Data to Big DataBalaBit
syslog-ng is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources, processes and filters them and forwards them to a destination. This session focuses on how syslog-ng parses important information from incoming messages, how to send this information to “big data” destinations, like HDFS, Kafka, ElasticSearch or MongoDB.
Abstract: syslog-ng is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources.
Raw log messages come in a variety of formats:
- lacking any structure most are usually just an almost proper English sentence with some variable parts in it, like user names or IP addresses.
- Fix table-like structure, like Apache access logs.
- A small minority of logs arrive in an already structured form: JSON.
Parsers in syslog-ng make it possible to extract important information from any of these messages and create name-value pairs.Once you have name-value pairs instead of raw log messages, you have many possibilities. On the syslog-ng side, you can use them for filtering, for example, to send an alert if the username is “root”. You can also use them in file names, or messages can be modified to facilitate log rotation or better suit applications processing the logs.
Parsing and preprocessing log messages also allows you to store them more effectively:
- you can send them to the destination (for example, ElasticSearch or MongoDB) in a format that can be easy to process (for example, JSON),
- you can filter irrelevant data, and forward only what is really needed,
- processing is off-loaded to very effective C code.
Finally you will learn about the “big data” destinations that syslog-ng supports, and how they benefit from message parsing:
- Hadoop Distributed File System ( HDFS ),
- Apache Kafka,
- ElasticSearch and Kibana, and
- MongoDB.
And if syslog-ng cannot already do something that you need, and you are not afraid of writing some code, you can learn about how language bindings of syslog-ng make it possible to add new destinations, not only in C, but also in Java, Lua, Perl, or Python.
Bio: Peter Czanik is community manager at Balabit, developers of syslog-ng. He helps distributions to maintain the syslog-ng package, follows bug trackers, helps users and talks regularly at conferences (FOSDEM, Libre Software Meeting, LOADays, Scale, etc.) In his limited free time he is interested in non-x86 architectures, and works on one of his PPC or ARM machines.
Get the most out of your security logs using syslog-ngPeter Czanik
Event logging is a central source of information for IT security. The syslog-ng application collects logs from many different sources, performs real-time log analysis by processing and filtering them, and finally it stores the logs or routes them for further analysis. This session focuses on how syslog-ng parses important information from incoming messages, enriches them with additional contextual information, and concludes with demonstrating how all of this can be used for alerting or for dashboards.
Scaling your logging infrastructure using syslog-ngPeter Czanik
This talk was presented at All Things Open: https://allthingsopen.org/talk/scaling-your-logging-infrastructure/
Event logging is important not only for IT security and operations, but also for business decisions. The syslog-ng application is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources, processes and filters them and finally it stores them or routes them for further analysis.
From this session you will learn (using examples from syslog-ng) why and how to parse important information from incoming messages, and how to route logs, feeding downstream systems using arbitrary formats. We will also discuss how the client – relay – server architecture can solve scalability problems. Also, I will present some of the recently introduced “Big Data” destinations of syslog-ng, which can help to scale your infrastructure even further.
gRPC is a modern high performance RPC (Remote Procedure Call) framework that can run in any environment. gRPC is based on HTTP/2, Protocol Buffers and other modern standard-based technologies. Together with excellent tooling, it helps you create high throughput, low latency, real-time services.
SCaLE 2016 - syslog-ng: From Raw Data to Big DataBalaBit
syslog-ng is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources, processes and filters them and forwards them to a destination. This session focuses on how syslog-ng parses important information from incoming messages, how to send this information to “big data” destinations, like HDFS, Kafka, ElasticSearch or MongoDB.
Abstract: syslog-ng is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources.
Raw log messages come in a variety of formats:
- lacking any structure most are usually just an almost proper English sentence with some variable parts in it, like user names or IP addresses.
- Fix table-like structure, like Apache access logs.
- A small minority of logs arrive in an already structured form: JSON.
Parsers in syslog-ng make it possible to extract important information from any of these messages and create name-value pairs.Once you have name-value pairs instead of raw log messages, you have many possibilities. On the syslog-ng side, you can use them for filtering, for example, to send an alert if the username is “root”. You can also use them in file names, or messages can be modified to facilitate log rotation or better suit applications processing the logs.
Parsing and preprocessing log messages also allows you to store them more effectively:
- you can send them to the destination (for example, ElasticSearch or MongoDB) in a format that can be easy to process (for example, JSON),
- you can filter irrelevant data, and forward only what is really needed,
- processing is off-loaded to very effective C code.
Finally you will learn about the “big data” destinations that syslog-ng supports, and how they benefit from message parsing:
- Hadoop Distributed File System ( HDFS ),
- Apache Kafka,
- ElasticSearch and Kibana, and
- MongoDB.
And if syslog-ng cannot already do something that you need, and you are not afraid of writing some code, you can learn about how language bindings of syslog-ng make it possible to add new destinations, not only in C, but also in Java, Lua, Perl, or Python.
Bio: Peter Czanik is community manager at Balabit, developers of syslog-ng. He helps distributions to maintain the syslog-ng package, follows bug trackers, helps users and talks regularly at conferences (FOSDEM, Libre Software Meeting, LOADays, Scale, etc.) In his limited free time he is interested in non-x86 architectures, and works on one of his PPC or ARM machines.
Get the most out of your security logs using syslog-ngPeter Czanik
Event logging is a central source of information for IT security. The syslog-ng application collects logs from many different sources, performs real-time log analysis by processing and filtering them, and finally it stores the logs or routes them for further analysis. This session focuses on how syslog-ng parses important information from incoming messages, enriches them with additional contextual information, and concludes with demonstrating how all of this can be used for alerting or for dashboards.
Scaling your logging infrastructure using syslog-ngPeter Czanik
This talk was presented at All Things Open: https://allthingsopen.org/talk/scaling-your-logging-infrastructure/
Event logging is important not only for IT security and operations, but also for business decisions. The syslog-ng application is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources, processes and filters them and finally it stores them or routes them for further analysis.
From this session you will learn (using examples from syslog-ng) why and how to parse important information from incoming messages, and how to route logs, feeding downstream systems using arbitrary formats. We will also discuss how the client – relay – server architecture can solve scalability problems. Also, I will present some of the recently introduced “Big Data” destinations of syslog-ng, which can help to scale your infrastructure even further.
gRPC is a modern high performance RPC (Remote Procedure Call) framework that can run in any environment. gRPC is based on HTTP/2, Protocol Buffers and other modern standard-based technologies. Together with excellent tooling, it helps you create high throughput, low latency, real-time services.
The Security library in VisualWorks went through sweeping changes recently. Main change is replacing native smalltalk implementations of various cryptographic algorithms with pluggable interfaces to external libraries, but also a complete rewrite of the SSL implementation to support all current versions of the protocol (SSL3.0 & TLS 1.0, 1.1 and 1.2). Introducing dependencies on external libraries can complicate deployment, however the resulting pluggability of implementation and perfomance boost we're getting in exchange should more then pay off in terms of widening the scope of potential applications, where the purely native implementation was simply not acceptable. In this talk we will survey these changes and discuss their impact and backward compatibility implications.
Hands-on tutorial on installation IPFS node and creation of smart contracts that use IPFS for data storage. As an example of IPFS usage in smart contracts, we create ERC-721 NFT that reference file in IPFS.
Tools and technologies used in this tutorial:
GCP https://console.cloud.google.com/home
ApiDapp https://apidapp.com/
Etherscan https://kovan.etherscan.io/
Solidity https://solidity.readthedocs.io/en/v0.6.1/
Open Zeppelin https://openzeppelin.com/contracts/
Python Refactoring with Rope and Traad – The rope library is a powerful tool for refactoring Python code, but to be truly useful it needs to be available to development environments. Traad is a tool which makes it simpler to integrate rope into nearly any tool by exposing a simple HTTP API. In this session we’ll look at how traad and rope work together, and we’ll see how traad integrates with at least one popular editor.
OpenSSH is a FREE version of the SSH connectivity tools that technical users of the Internet rely on.
This talk will explain the most interesting features of ssh and some info about future developments.
2015. Libre Software Meeting - syslog-ng: from log collection to processing a...BalaBit
Peter Czanik: syslog-ng - from log collection to processing and infomation extraction
2015. Libre Software Meeting
After a short introduction to system logging, we will show how the current log messages look like, and what the problem is with this free text format. Next, we will introduce you the powerful concept of name-value pairs, and how you can extract useful information from your logs by parsing log messages into name-value pairs. Next we will demonstrate the flexibility of syslog-ng’s message parsers (patterndb, csv and JSON parsers), and show you how to create patterns using a text editor or a GUI. This can also be used to overwrite sensitive information due to privacy regulations. At the end, you will learn about the Perl/Python/Lua/Java bindings of syslog-ng Open Source Edition, how value pairs can be passed to them, and some reference applications written for syslog-ng.
The Security library in VisualWorks went through sweeping changes recently. Main change is replacing native smalltalk implementations of various cryptographic algorithms with pluggable interfaces to external libraries, but also a complete rewrite of the SSL implementation to support all current versions of the protocol (SSL3.0 & TLS 1.0, 1.1 and 1.2). Introducing dependencies on external libraries can complicate deployment, however the resulting pluggability of implementation and perfomance boost we're getting in exchange should more then pay off in terms of widening the scope of potential applications, where the purely native implementation was simply not acceptable. In this talk we will survey these changes and discuss their impact and backward compatibility implications.
Hands-on tutorial on installation IPFS node and creation of smart contracts that use IPFS for data storage. As an example of IPFS usage in smart contracts, we create ERC-721 NFT that reference file in IPFS.
Tools and technologies used in this tutorial:
GCP https://console.cloud.google.com/home
ApiDapp https://apidapp.com/
Etherscan https://kovan.etherscan.io/
Solidity https://solidity.readthedocs.io/en/v0.6.1/
Open Zeppelin https://openzeppelin.com/contracts/
Python Refactoring with Rope and Traad – The rope library is a powerful tool for refactoring Python code, but to be truly useful it needs to be available to development environments. Traad is a tool which makes it simpler to integrate rope into nearly any tool by exposing a simple HTTP API. In this session we’ll look at how traad and rope work together, and we’ll see how traad integrates with at least one popular editor.
OpenSSH is a FREE version of the SSH connectivity tools that technical users of the Internet rely on.
This talk will explain the most interesting features of ssh and some info about future developments.
2015. Libre Software Meeting - syslog-ng: from log collection to processing a...BalaBit
Peter Czanik: syslog-ng - from log collection to processing and infomation extraction
2015. Libre Software Meeting
After a short introduction to system logging, we will show how the current log messages look like, and what the problem is with this free text format. Next, we will introduce you the powerful concept of name-value pairs, and how you can extract useful information from your logs by parsing log messages into name-value pairs. Next we will demonstrate the flexibility of syslog-ng’s message parsers (patterndb, csv and JSON parsers), and show you how to create patterns using a text editor or a GUI. This can also be used to overwrite sensitive information due to privacy regulations. At the end, you will learn about the Perl/Python/Lua/Java bindings of syslog-ng Open Source Edition, how value pairs can be passed to them, and some reference applications written for syslog-ng.
Big Data, Data Lake, Fast Data - Dataserialiation-FormatsGuido Schmutz
The concept of "Data Lake" is in everyone's mind today. The idea of storing all the data that accumulates in a company in a central location and making it available sounds very interesting at first. But Data Lake can quickly turn from a clear, beautiful mountain lake into a huge pond, especially if it is inexpertly entrusted with all the source data formats that are common in today's enterprises, such as XML, JSON, CSV or unstructured text data. Who, after some time, still has an overview of which data, which format and how they have developed over different versions? Anyone who wants to help themselves from the Data Lake must ask themselves the same questions over and over again: what information is provided, what data types do they have and how has the content changed over time?
Data serialization frameworks such as Apache Avro and Google Protocol Buffer (Protobuf), which enable platform-independent data modeling and data storage, can help. This talk will discuss the possibilities of Avro and Protobuf and show how they can be used in the context of a data lake and what advantages can be achieved. The support on Avro and Protobuf by Big Data and Fast Data platforms is also a topic.
Managing Your Security Logs with ElasticsearchVic Hargrave
The ELK stack (Elasticsearch-Logstash-Kibana) provides a cost effective alternative to commercial SIEMs for ingesting and managing OSSEC alert logs. This presentation will show you how to construct a low cost SIEM based on ELK that rivals the capabilties of commercials SIEMs.
This is the talk I have given on Fedora Developer's Conference 2014 in Brno. It provides insight into the security features we added to rsyslog v7, integration into systemd journal, enhancements of the v8 engine and a glimpse at how to write rsyslog plugins in languages other than C.
Apache Drill is new Apache incubator project. It's goal is to provide a distributed system for interactive analysis of large-scale datasets. Inspired by Google's Dremel technology, it aims to process trillions of records in seconds. We will cover the goals of Apache Drill, its use cases and how it relates to Hadoop, MongoDB and other large-scale distributed systems. We'll also talk about details of the architecture, points of extensibility, data flow and our first query languages (DrQL and SQL).
Application Logging in the 21st century - 2014.keyTim Bunce
Slides for my talk at the Austrian Perl Workshop in Salzburg on October 10th.
A video of the talk can be found at https://www.youtube.com/watch?v=4Qj-_eimGuE
You Can't Correlate what you don't have - ArcSight Protect 2011Scott Carlson
In this presentation we discuss gathering data with syslog-ng in order to properly feed your SIEM system such as ArcSight ESM. This presentation is from HP/ArcSight Protect 2011.
OpenIDM - Flexible Provisioning Platform - April 28 WebinarForgeRock
Identity Management requires powerful extensibility for handling lifecycle management use cases specific to each business. Legacy identity management solutions handled this poorly, using proprietary scripting languages that were painful and required specialized knowledge. ForgeRock designed OpenIDM with rapid extensibility in mind.
In this webinar, we will provide an overview of OpenIDM, explain the power of OpenIDM's javascript / groovy scripting mechanism and demonstrate how it can be used to generate a privilege user management script with less than 60 lines of javascript code. The sample code will also be made available post webinar for developers that want to play.
Apache Solr on Hadoop is enabling organizations to collect, process and search larger, more varied data. Apache Spark is is making a large impact across the industry, changing the way we think about batch processing and replacing MapReduce in many cases. But how can production users easily migrate ingestion of HDFS data into Solr from MapReduce to Spark? How can they update and delete existing documents in Solr at scale? And how can they easily build flexible data ingestion pipelines? Cloudera Search Software Engineer Wolfgang Hoschek will present an architecture and solution to this problem. How was Apache Solr, Spark, Crunch, and Morphlines integrated to allow for scalable and flexible ingestion of HDFS data into Solr? What are the solved problems and what's still to come? Join us for an exciting discussion on this new technology.
OpenShift Origin Community Day (Boston) Writing Cartridges V2 by Jhon Honce Diane Mueller
Presenters: Jhon Honce
Cartridges allow developers to provide services running on top of the Red Hat OpenShift Platform-as-a-Service (PaaS). OpenShift already provides cartridges for numerous web application frameworks and databases. Writing your own cartridges allows you to customize or enhance an existing service, or provide new services. In this session, the presenter will discuss best practices for cartridge development and the latest changes in the OpenShift cartridge support.
* Latest changes made in the platform to ease cartridge development
* OpenShift Cartridges vs. plugins
* Outline for development of a new cartridge
* Customization of existing cartridges
* Quickstarts: leveraging a cartridge or cartridges to provide a complete application
OpenShift Origin Community Day (Boston) Extending OpenShift Origin: Build You...OpenShift Origin
Extending OpenShift Origin: Build Your Own Cartridge
Presenters: Jhon Honce
Cartridges allow developers to provide services running on top of the Red Hat OpenShift Platform-as-a-Service (PaaS). OpenShift already provides cartridges for numerous web application frameworks and databases. Writing your own cartridges allows you to customize or enhance an existing service, or provide new services. In this session, the presenter will discuss best practices for cartridge development and the latest changes in the OpenShift cartridge support.
* Latest changes made in the platform to ease cartridge development
* OpenShift Cartridges vs. plugins
* Outline for development of a new cartridge
* Customization of existing cartridges
* Quickstarts: leveraging a cartridge or cartridges to provide a complete application
NIAS 2015 - The value add of open source for innovationBalaBit
(Balázs Scheidler, co-founder and CTO, BalaBit)
As a long term member of the Open Source community, I believe that the Open Source development model creates a great context for innovation to happen. In the open source world, collaboration and sharing are key principles. These principles put the problem to be solved in focus and tear down
organizational boundaries. An Open Source project is a space where the best engineers from multiple competing organizations work as a team on solving a common goal. This space and the direct connection to users boosts engineer motivation, creating trust and a virtuous circle that results in fast iterations: creating layers upon layers of work yielding a great product at a pace that is unrealistic in a proprietary software development setting. We can also see that the same values and principles start happening outside of the software realm: Wikipedia, Creative Commons and the Maker community confirms the approach works in other fields, which shows that it can be adapted to further situations to improve efficiency and innovation.
Les Assises 2015 - Why people are the most important aspect of IT security?BalaBit
Balázs Scheidler, co-founder and CTO of BalaBit holds a presentation about the importance of privileged users in IT security. He introduces BalaBit's approach to people-centric security - people centric security is a strategic approach to information security that emphasizes individual accountability and trust. It de-emphasizes restrictive, preventive security controls, while the monitoring of user activities is a fundamental element of people centric security.
Mr. Scheidler showcases how cooperates Blindspotter, BalaBit's UBA solution with its Privileged Activity Monitoring tool, Shell Control Box, and how does they provide an effective defense against Advanced Persistent Threats. A live demo of how an APT attack would be prevented will be also part of the presentation.
A recent eCSI survey reveals that nearly a quarter of IT professionals use firewalls as their only protection against malicious insiders and targeted attacks, which is completely ineffective for that purpose.
Hogyan maradj egészséges irodai munka mellett?BalaBit
Hogy kiknek szól ez a kiadvány? Amatőröknek. Pályakezdőknek. Újrakezdőknek. Sohaelnemkezdőknek. Mármint egészség fronton. Fájós háttal, úszógumival a derekukon munkában úszó sorstársainknak.
Hogy kiknek nem szól? Profiknak nem szól, mert ahhoz szándékosan kevés. Misztikus-egzotikus csodaprogramokat habzsoló divat-egészségeseknek sem szól, mert nincs könnyű út.
Csak egy út van.
Az alábbi pár oldalon összeszedtük a legfontosabb tanácsokat és megtámogattuk őket azokkal a tényeket, ami biológiából, anatómiából, kémiából stb... következik. Hiszünk benne, hogy a megértés segíti a tudás alkalmazását.
Log messages can be used to detect security incidents, operational problems, and other issues like policy violations, and are useful in auditing and forensics situations.
From this white paper you can learn the advantages of using the syslog-ng Store Box logserver appliance to collect, store, and manage system log (syslog) and eventlog messages for policy compliance.
Kontrolle und revisionssichere Auditierung privilegierter IT-ZugriffeBalaBit
In einem Unternehmen gibt es meist eine Vielzahl unbekannter privilegierter Benutzer: Systemadministratoren, Benutzer mit Zugriff auf vertrauliche Inhalte – besonders in IT-Architekturen, die mehrere Altsysteme enthalten.
LOADays 2015 - syslog-ng - from log collection to processing and infomation extraction
1. syslog-ng: from log collection to
processing and information extraction
LOADays 2015
Peter Czanik / BalaBit
2. 2
About me
■ Peter Czanik from Hungary
■ Community manager at BalaBit: syslog-ng upstream
■ Doing syslog-ng packaging, support, advocating
■ BalaBit is an IT security company with development HQ in Budapest,
Hungary
■ Over 200 employees: the majority are engineers
3. 3
Topics
■ What is syslog-ng
■ Basic syslog-ng configuration
■ The importance of structured log messages
■ Message parsing
■ Creating patterns for PatternDB
■ Language bindings
■ syslog-ng central management
4. 4
Syslog → syslog-ng
■ Logging: recording events
■ Jan 14 11:38:48 linux-0jbu sshd[7716]: Accepted publickey for root from
127.0.0.1 port 48806 ssh2
■ syslog-ng: enhanced log daemon, with a focus on central log collection,
supporting a wide range of input and output methods with a flexible
configuration language
8. 8
syslog-ng: destinations
■ Traditional file and UDP/TCP/TLS destinations
■ SQL and NoSQL destinations (mysql, mongodb)
■ Visualization (graphite)
■ Alerting (riemann)
■ Message queuing (RabbitMQ, ZeroMQ)
■ Hadoop, Elasticsearch, Kafka and many more
9. 9
Configuration
■ “Don't Panic”
■ Simple and logical, even if looks difficult
■ Pipeline model:
□ Many different building blocks (sources, destinations, filters, parsers, etc.)
□ Connected using “log” statements into a pipeline
■ Sample config from Fedora
15. 15
Free-form log messages
■ Most log messages are: date + hostname + text
Mar 11 13:37:56 linux-6965 sshd[4547]: Accepted keyboard-
interactive/pam for root from 127.0.0.1 port 46048 ssh2
■ Text = English sentence with some variable parts
■ Easy to read by a human
16. 16
Why it does not scale
■ Information is presented differently by each application
■ Few logs (workstation) → easy to find information
■ Many logs (server) → difficult to find information
■ Difficult to process them with scripts
17. 17
Solution: structured logging
■ Events represented as name-value pairs
■ Example: an ssh login:
□ source_ip=192.168.123.45
□ app=sshd
□ user=root
■ Parsers in syslog-ng can turn unstructured and some structured data (csv,
JSON) into name value pairs
■ syslog-ng: name-value pairs inside
□ Date, facility, priority, program name, pid, etc.
■ Templates: use name-value pairs for custom file names or messages
20. 20
PatternDB parser
■ PatternDB message parser:
□ Can extract useful information from unstructured messages into name-value
pairs
□ Add status fields based on message text
□ Message classification (like LogCheck)
■ Needs XML describing log messages
■ Example: an ssh login failure:
□ user=root, source_ip=192.168.123.45, action=login, status=failure
□ classified as “violation”
22. 22
Creating patterns for syslog-ng: editor
■ Some sample patterns available:
□ https://github.com/balabit/syslog-ng-patterndb
■ Use an XML editor or text editor with syntax highlighting
■ Use “pdbtool” to
□ test, debug
□ merge
□ convert
patterns
23. 23
Creating patterns for syslog-ng: Puppet
■ More friendly format (especially if you use Puppet :-) )
■ https://github.com/ccin2p3/puppet-patterndb
■ Use “pdbtool” as usual
patterndb::simple::ruleset { 'myruleset':
id => '9586b525-826e-4c2d-b74f-381039cf470c',
patterns => [ 'sshd' ],
pubdate => '2014-03-24',
rules => [
{
id => 'd69bd1ed-17ff-4667-8ea4-087170cbceeb',
patterns => ['Successful login for user @QSTRING:user:"@ using method @QSTRING:method:"@']
}
]
}
24. 24
Creating patterns for syslog-ng: GUI
■ This is a work in progress
■ Finds patterns automagically from similar lines
■ Fields can be edited and named
■ Results can be verified
27. 27
Anonymizing messages
■ Many regulations about what can be logged
□ PCI-DSS: credit card numbers
□ Europe: IP addresses, user names
■ Locating sensitive information:
□ Regular expressions: slow, works also in unknown logs
□ Patterndb: fast, only in known log messages
■ Anonymizing:
□ Overwrite it with constant
□ Overwrite it with a hash of the original
28. 28
Language bindings in syslog-ng
■ The primary language of syslog-ng is C:
□ High performance: processes a lot more EPS than interpreted languages
■ Not everything is implemented in C
■ Rapid prototyping is easier in interpreted languages
■ Lua / Perl / Python / Java destinations, Lua monitoring source
□ Embedded interpreter
□ Message or full range of name value pairs can be passed
■ Java/Python moving from incubator to core in 3.7
29. 29
ElasticSearch through Java destination
■ Syslog-ng 3.7 beta has Java destination (originally in the incubator)
■ https://github.com/juhaszviktor/ESDestination
destination d_es {
java(
class_path("/usr/local/ESDestination.jar:/usr/share/elasticsearch/lib/*.jar")
class_name("org.syslog_ng.elasticsearch.ElasticSearchDestination")
option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}")
option("type", "test")
option("cluster", "syslog-ng")
option("flush_limit", "100")
option("custom_id", "$RCPTID")
);
};
30. 30
Central syslog-ng management
■ modules for Puppet, Salt and Ansible
■ Puppet is the most tested with thousands of machines
■ https://github.com/ihrwein/puppet-syslog_ng
■ Features:
□ Installs syslog-ng and sub-modules
□ Can configure syslog-ng with minimal limitations
33. 33
Interactive syslog-ng
■ See which path a log message takes inside syslog-ng
■ Stop at break points
■ Show current state of macros
■ Built-in help and tab completion
■ Initial commit in syslog-ng 3.7 (beta)
■ Feedback is very welcome!
34. 34
Questions? (and some answers)
■ Questions?
■ Some useful syslog-ng resources:
□ syslog-ng: http://syslog-ng.org/
□ ELSA (log analysis based on syslog-ng's patterndb):
http://code.google.com/p/enterprise-log-search-and-archive/
□ Alerting: http://devops.com/features/guide-modern-monitoring-alerting/
□ Mailing list: https://lists.balabit.hu/pipermail/syslog-ng/
□ My blog: http://czanik.blogs.balabit.com/
□ My e-mail: czanik@balabit.hu