Peter Czanik: syslog-ng - from log collection to processing and infomation extraction
2015. Libre Software Meeting
After a short introduction to system logging, we will show how the current log messages look like, and what the problem is with this free text format. Next, we will introduce you the powerful concept of name-value pairs, and how you can extract useful information from your logs by parsing log messages into name-value pairs. Next we will demonstrate the flexibility of syslog-ng’s message parsers (patterndb, csv and JSON parsers), and show you how to create patterns using a text editor or a GUI. This can also be used to overwrite sensitive information due to privacy regulations. At the end, you will learn about the Perl/Python/Lua/Java bindings of syslog-ng Open Source Edition, how value pairs can be passed to them, and some reference applications written for syslog-ng.
SCaLE 2016 - syslog-ng: From Raw Data to Big DataBalaBit
syslog-ng is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources, processes and filters them and forwards them to a destination. This session focuses on how syslog-ng parses important information from incoming messages, how to send this information to “big data” destinations, like HDFS, Kafka, ElasticSearch or MongoDB.
Abstract: syslog-ng is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources.
Raw log messages come in a variety of formats:
- lacking any structure most are usually just an almost proper English sentence with some variable parts in it, like user names or IP addresses.
- Fix table-like structure, like Apache access logs.
- A small minority of logs arrive in an already structured form: JSON.
Parsers in syslog-ng make it possible to extract important information from any of these messages and create name-value pairs.Once you have name-value pairs instead of raw log messages, you have many possibilities. On the syslog-ng side, you can use them for filtering, for example, to send an alert if the username is “root”. You can also use them in file names, or messages can be modified to facilitate log rotation or better suit applications processing the logs.
Parsing and preprocessing log messages also allows you to store them more effectively:
- you can send them to the destination (for example, ElasticSearch or MongoDB) in a format that can be easy to process (for example, JSON),
- you can filter irrelevant data, and forward only what is really needed,
- processing is off-loaded to very effective C code.
Finally you will learn about the “big data” destinations that syslog-ng supports, and how they benefit from message parsing:
- Hadoop Distributed File System ( HDFS ),
- Apache Kafka,
- ElasticSearch and Kibana, and
- MongoDB.
And if syslog-ng cannot already do something that you need, and you are not afraid of writing some code, you can learn about how language bindings of syslog-ng make it possible to add new destinations, not only in C, but also in Java, Lua, Perl, or Python.
Bio: Peter Czanik is community manager at Balabit, developers of syslog-ng. He helps distributions to maintain the syslog-ng package, follows bug trackers, helps users and talks regularly at conferences (FOSDEM, Libre Software Meeting, LOADays, Scale, etc.) In his limited free time he is interested in non-x86 architectures, and works on one of his PPC or ARM machines.
Get the most out of your security logs using syslog-ngPeter Czanik
Event logging is a central source of information for IT security. The syslog-ng application collects logs from many different sources, performs real-time log analysis by processing and filtering them, and finally it stores the logs or routes them for further analysis. This session focuses on how syslog-ng parses important information from incoming messages, enriches them with additional contextual information, and concludes with demonstrating how all of this can be used for alerting or for dashboards.
Scaling your logging infrastructure using syslog-ngPeter Czanik
This talk was presented at All Things Open: https://allthingsopen.org/talk/scaling-your-logging-infrastructure/
Event logging is important not only for IT security and operations, but also for business decisions. The syslog-ng application is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources, processes and filters them and finally it stores them or routes them for further analysis.
From this session you will learn (using examples from syslog-ng) why and how to parse important information from incoming messages, and how to route logs, feeding downstream systems using arbitrary formats. We will also discuss how the client – relay – server architecture can solve scalability problems. Also, I will present some of the recently introduced “Big Data” destinations of syslog-ng, which can help to scale your infrastructure even further.
gRPC is a modern high performance RPC (Remote Procedure Call) framework that can run in any environment. gRPC is based on HTTP/2, Protocol Buffers and other modern standard-based technologies. Together with excellent tooling, it helps you create high throughput, low latency, real-time services.
SCaLE 2016 - syslog-ng: From Raw Data to Big DataBalaBit
syslog-ng is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources, processes and filters them and forwards them to a destination. This session focuses on how syslog-ng parses important information from incoming messages, how to send this information to “big data” destinations, like HDFS, Kafka, ElasticSearch or MongoDB.
Abstract: syslog-ng is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources.
Raw log messages come in a variety of formats:
- lacking any structure most are usually just an almost proper English sentence with some variable parts in it, like user names or IP addresses.
- Fix table-like structure, like Apache access logs.
- A small minority of logs arrive in an already structured form: JSON.
Parsers in syslog-ng make it possible to extract important information from any of these messages and create name-value pairs.Once you have name-value pairs instead of raw log messages, you have many possibilities. On the syslog-ng side, you can use them for filtering, for example, to send an alert if the username is “root”. You can also use them in file names, or messages can be modified to facilitate log rotation or better suit applications processing the logs.
Parsing and preprocessing log messages also allows you to store them more effectively:
- you can send them to the destination (for example, ElasticSearch or MongoDB) in a format that can be easy to process (for example, JSON),
- you can filter irrelevant data, and forward only what is really needed,
- processing is off-loaded to very effective C code.
Finally you will learn about the “big data” destinations that syslog-ng supports, and how they benefit from message parsing:
- Hadoop Distributed File System ( HDFS ),
- Apache Kafka,
- ElasticSearch and Kibana, and
- MongoDB.
And if syslog-ng cannot already do something that you need, and you are not afraid of writing some code, you can learn about how language bindings of syslog-ng make it possible to add new destinations, not only in C, but also in Java, Lua, Perl, or Python.
Bio: Peter Czanik is community manager at Balabit, developers of syslog-ng. He helps distributions to maintain the syslog-ng package, follows bug trackers, helps users and talks regularly at conferences (FOSDEM, Libre Software Meeting, LOADays, Scale, etc.) In his limited free time he is interested in non-x86 architectures, and works on one of his PPC or ARM machines.
Get the most out of your security logs using syslog-ngPeter Czanik
Event logging is a central source of information for IT security. The syslog-ng application collects logs from many different sources, performs real-time log analysis by processing and filtering them, and finally it stores the logs or routes them for further analysis. This session focuses on how syslog-ng parses important information from incoming messages, enriches them with additional contextual information, and concludes with demonstrating how all of this can be used for alerting or for dashboards.
Scaling your logging infrastructure using syslog-ngPeter Czanik
This talk was presented at All Things Open: https://allthingsopen.org/talk/scaling-your-logging-infrastructure/
Event logging is important not only for IT security and operations, but also for business decisions. The syslog-ng application is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources, processes and filters them and finally it stores them or routes them for further analysis.
From this session you will learn (using examples from syslog-ng) why and how to parse important information from incoming messages, and how to route logs, feeding downstream systems using arbitrary formats. We will also discuss how the client – relay – server architecture can solve scalability problems. Also, I will present some of the recently introduced “Big Data” destinations of syslog-ng, which can help to scale your infrastructure even further.
gRPC is a modern high performance RPC (Remote Procedure Call) framework that can run in any environment. gRPC is based on HTTP/2, Protocol Buffers and other modern standard-based technologies. Together with excellent tooling, it helps you create high throughput, low latency, real-time services.
The Security library in VisualWorks went through sweeping changes recently. Main change is replacing native smalltalk implementations of various cryptographic algorithms with pluggable interfaces to external libraries, but also a complete rewrite of the SSL implementation to support all current versions of the protocol (SSL3.0 & TLS 1.0, 1.1 and 1.2). Introducing dependencies on external libraries can complicate deployment, however the resulting pluggability of implementation and perfomance boost we're getting in exchange should more then pay off in terms of widening the scope of potential applications, where the purely native implementation was simply not acceptable. In this talk we will survey these changes and discuss their impact and backward compatibility implications.
Hands-on tutorial on installation IPFS node and creation of smart contracts that use IPFS for data storage. As an example of IPFS usage in smart contracts, we create ERC-721 NFT that reference file in IPFS.
Tools and technologies used in this tutorial:
GCP https://console.cloud.google.com/home
ApiDapp https://apidapp.com/
Etherscan https://kovan.etherscan.io/
Solidity https://solidity.readthedocs.io/en/v0.6.1/
Open Zeppelin https://openzeppelin.com/contracts/
OpenSSH is a FREE version of the SSH connectivity tools that technical users of the Internet rely on.
This talk will explain the most interesting features of ssh and some info about future developments.
Python Refactoring with Rope and Traad – The rope library is a powerful tool for refactoring Python code, but to be truly useful it needs to be available to development environments. Traad is a tool which makes it simpler to integrate rope into nearly any tool by exposing a simple HTTP API. In this session we’ll look at how traad and rope work together, and we’ll see how traad integrates with at least one popular editor.
ArcBlock Technical Learning Series Presents IPFS.
If there's a missing piece in current blockchain stack, that'll be a decentralized, public verifiable file system. Ideally before decentralizing computing, we shall decentralize the data. IPFS filled in this area, and it has a great potential to push web to the true web3 - decentralized web. This talk will talk about what problem IPFS is trying to solve, how it solves the problem, and how to use IPFS in our applications.
https://www.arcblock.io
https://hack.arcblock.io/learning
LOADays 2015 - syslog-ng - from log collection to processing and infomation e...BalaBit
Peter Czanik: syslog-ng - from log collection to processing and infomation extraction
LOADays 2015.
After a short introduction to system logging, we will show how the current log messages look like, and what the problem is with this free text format. Next, we will introduce you the powerful concept of name-value pairs, and how you can extract useful information from your logs by parsing log messages into name-value pairs. Next we will demonstrate the flexibility of syslog-ng’s message parsers (patterndb, csv and JSON parsers), and show you how to create patterns using a text editor or a GUI. This can also be used to overwrite sensitive information due to privacy regulations. At the end, you will learn about the Perl/Python/Lua/Java bindings of syslog-ng Open Source Edition, how value pairs can be passed to them, and some reference applications written for syslog-ng.
The Security library in VisualWorks went through sweeping changes recently. Main change is replacing native smalltalk implementations of various cryptographic algorithms with pluggable interfaces to external libraries, but also a complete rewrite of the SSL implementation to support all current versions of the protocol (SSL3.0 & TLS 1.0, 1.1 and 1.2). Introducing dependencies on external libraries can complicate deployment, however the resulting pluggability of implementation and perfomance boost we're getting in exchange should more then pay off in terms of widening the scope of potential applications, where the purely native implementation was simply not acceptable. In this talk we will survey these changes and discuss their impact and backward compatibility implications.
Hands-on tutorial on installation IPFS node and creation of smart contracts that use IPFS for data storage. As an example of IPFS usage in smart contracts, we create ERC-721 NFT that reference file in IPFS.
Tools and technologies used in this tutorial:
GCP https://console.cloud.google.com/home
ApiDapp https://apidapp.com/
Etherscan https://kovan.etherscan.io/
Solidity https://solidity.readthedocs.io/en/v0.6.1/
Open Zeppelin https://openzeppelin.com/contracts/
OpenSSH is a FREE version of the SSH connectivity tools that technical users of the Internet rely on.
This talk will explain the most interesting features of ssh and some info about future developments.
Python Refactoring with Rope and Traad – The rope library is a powerful tool for refactoring Python code, but to be truly useful it needs to be available to development environments. Traad is a tool which makes it simpler to integrate rope into nearly any tool by exposing a simple HTTP API. In this session we’ll look at how traad and rope work together, and we’ll see how traad integrates with at least one popular editor.
ArcBlock Technical Learning Series Presents IPFS.
If there's a missing piece in current blockchain stack, that'll be a decentralized, public verifiable file system. Ideally before decentralizing computing, we shall decentralize the data. IPFS filled in this area, and it has a great potential to push web to the true web3 - decentralized web. This talk will talk about what problem IPFS is trying to solve, how it solves the problem, and how to use IPFS in our applications.
https://www.arcblock.io
https://hack.arcblock.io/learning
LOADays 2015 - syslog-ng - from log collection to processing and infomation e...BalaBit
Peter Czanik: syslog-ng - from log collection to processing and infomation extraction
LOADays 2015.
After a short introduction to system logging, we will show how the current log messages look like, and what the problem is with this free text format. Next, we will introduce you the powerful concept of name-value pairs, and how you can extract useful information from your logs by parsing log messages into name-value pairs. Next we will demonstrate the flexibility of syslog-ng’s message parsers (patterndb, csv and JSON parsers), and show you how to create patterns using a text editor or a GUI. This can also be used to overwrite sensitive information due to privacy regulations. At the end, you will learn about the Perl/Python/Lua/Java bindings of syslog-ng Open Source Edition, how value pairs can be passed to them, and some reference applications written for syslog-ng.
Big Data, Data Lake, Fast Data - Dataserialiation-FormatsGuido Schmutz
The concept of "Data Lake" is in everyone's mind today. The idea of storing all the data that accumulates in a company in a central location and making it available sounds very interesting at first. But Data Lake can quickly turn from a clear, beautiful mountain lake into a huge pond, especially if it is inexpertly entrusted with all the source data formats that are common in today's enterprises, such as XML, JSON, CSV or unstructured text data. Who, after some time, still has an overview of which data, which format and how they have developed over different versions? Anyone who wants to help themselves from the Data Lake must ask themselves the same questions over and over again: what information is provided, what data types do they have and how has the content changed over time?
Data serialization frameworks such as Apache Avro and Google Protocol Buffer (Protobuf), which enable platform-independent data modeling and data storage, can help. This talk will discuss the possibilities of Avro and Protobuf and show how they can be used in the context of a data lake and what advantages can be achieved. The support on Avro and Protobuf by Big Data and Fast Data platforms is also a topic.
Managing Your Security Logs with ElasticsearchVic Hargrave
The ELK stack (Elasticsearch-Logstash-Kibana) provides a cost effective alternative to commercial SIEMs for ingesting and managing OSSEC alert logs. This presentation will show you how to construct a low cost SIEM based on ELK that rivals the capabilties of commercials SIEMs.
Application Logging in the 21st century - 2014.keyTim Bunce
Slides for my talk at the Austrian Perl Workshop in Salzburg on October 10th.
A video of the talk can be found at https://www.youtube.com/watch?v=4Qj-_eimGuE
This is the talk I have given on Fedora Developer's Conference 2014 in Brno. It provides insight into the security features we added to rsyslog v7, integration into systemd journal, enhancements of the v8 engine and a glimpse at how to write rsyslog plugins in languages other than C.
OpenShift Origin Community Day (Boston) Writing Cartridges V2 by Jhon Honce Diane Mueller
Presenters: Jhon Honce
Cartridges allow developers to provide services running on top of the Red Hat OpenShift Platform-as-a-Service (PaaS). OpenShift already provides cartridges for numerous web application frameworks and databases. Writing your own cartridges allows you to customize or enhance an existing service, or provide new services. In this session, the presenter will discuss best practices for cartridge development and the latest changes in the OpenShift cartridge support.
* Latest changes made in the platform to ease cartridge development
* OpenShift Cartridges vs. plugins
* Outline for development of a new cartridge
* Customization of existing cartridges
* Quickstarts: leveraging a cartridge or cartridges to provide a complete application
OpenShift Origin Community Day (Boston) Extending OpenShift Origin: Build You...OpenShift Origin
Extending OpenShift Origin: Build Your Own Cartridge
Presenters: Jhon Honce
Cartridges allow developers to provide services running on top of the Red Hat OpenShift Platform-as-a-Service (PaaS). OpenShift already provides cartridges for numerous web application frameworks and databases. Writing your own cartridges allows you to customize or enhance an existing service, or provide new services. In this session, the presenter will discuss best practices for cartridge development and the latest changes in the OpenShift cartridge support.
* Latest changes made in the platform to ease cartridge development
* OpenShift Cartridges vs. plugins
* Outline for development of a new cartridge
* Customization of existing cartridges
* Quickstarts: leveraging a cartridge or cartridges to provide a complete application
You Can't Correlate what you don't have - ArcSight Protect 2011Scott Carlson
In this presentation we discuss gathering data with syslog-ng in order to properly feed your SIEM system such as ArcSight ESM. This presentation is from HP/ArcSight Protect 2011.
Apache Solr on Hadoop is enabling organizations to collect, process and search larger, more varied data. Apache Spark is is making a large impact across the industry, changing the way we think about batch processing and replacing MapReduce in many cases. But how can production users easily migrate ingestion of HDFS data into Solr from MapReduce to Spark? How can they update and delete existing documents in Solr at scale? And how can they easily build flexible data ingestion pipelines? Cloudera Search Software Engineer Wolfgang Hoschek will present an architecture and solution to this problem. How was Apache Solr, Spark, Crunch, and Morphlines integrated to allow for scalable and flexible ingestion of HDFS data into Solr? What are the solved problems and what's still to come? Join us for an exciting discussion on this new technology.
You can only monitor systems that you know!
GLPI is a very successful open source ITSM solution, the project follows a modular approach and can therefore be extended by many very useful plugins. And yes … GLPI is mainly “French” !
In this very short introduction, I’ll will give you a rapid overview how to:
– automate your IT inventory to manage pc’s, servers, vm’s, vmware, …
– add printers and network components via “snmp”
– add special assets like databases, appliances, URL’s, lines, racks, datacenters…
– add additional information’s to all this components
– add people from your LDAP / AD
– add plugins to GLPI
– build reports
– import / export your data
– handle tickets, problems, changes, or projects
In my second presentation “Monitoring @ G&D ” I will later show you how we’ve automated our monitoring with the help of GLPI, some db view’s and python scripts.
Learn about structured logging with rsyslog and how it can be used to do actual format conversions. Include config samples for Linux and Windows log sources.
Similar to 2015. Libre Software Meeting - syslog-ng: from log collection to processing and information extraction (20)
NIAS 2015 - The value add of open source for innovationBalaBit
(Balázs Scheidler, co-founder and CTO, BalaBit)
As a long term member of the Open Source community, I believe that the Open Source development model creates a great context for innovation to happen. In the open source world, collaboration and sharing are key principles. These principles put the problem to be solved in focus and tear down
organizational boundaries. An Open Source project is a space where the best engineers from multiple competing organizations work as a team on solving a common goal. This space and the direct connection to users boosts engineer motivation, creating trust and a virtuous circle that results in fast iterations: creating layers upon layers of work yielding a great product at a pace that is unrealistic in a proprietary software development setting. We can also see that the same values and principles start happening outside of the software realm: Wikipedia, Creative Commons and the Maker community confirms the approach works in other fields, which shows that it can be adapted to further situations to improve efficiency and innovation.
Les Assises 2015 - Why people are the most important aspect of IT security?BalaBit
Balázs Scheidler, co-founder and CTO of BalaBit holds a presentation about the importance of privileged users in IT security. He introduces BalaBit's approach to people-centric security - people centric security is a strategic approach to information security that emphasizes individual accountability and trust. It de-emphasizes restrictive, preventive security controls, while the monitoring of user activities is a fundamental element of people centric security.
Mr. Scheidler showcases how cooperates Blindspotter, BalaBit's UBA solution with its Privileged Activity Monitoring tool, Shell Control Box, and how does they provide an effective defense against Advanced Persistent Threats. A live demo of how an APT attack would be prevented will be also part of the presentation.
A recent eCSI survey reveals that nearly a quarter of IT professionals use firewalls as their only protection against malicious insiders and targeted attacks, which is completely ineffective for that purpose.
Hogyan maradj egészséges irodai munka mellett?BalaBit
Hogy kiknek szól ez a kiadvány? Amatőröknek. Pályakezdőknek. Újrakezdőknek. Sohaelnemkezdőknek. Mármint egészség fronton. Fájós háttal, úszógumival a derekukon munkában úszó sorstársainknak.
Hogy kiknek nem szól? Profiknak nem szól, mert ahhoz szándékosan kevés. Misztikus-egzotikus csodaprogramokat habzsoló divat-egészségeseknek sem szól, mert nincs könnyű út.
Csak egy út van.
Az alábbi pár oldalon összeszedtük a legfontosabb tanácsokat és megtámogattuk őket azokkal a tényeket, ami biológiából, anatómiából, kémiából stb... következik. Hiszünk benne, hogy a megértés segíti a tudás alkalmazását.
Log messages can be used to detect security incidents, operational problems, and other issues like policy violations, and are useful in auditing and forensics situations.
From this white paper you can learn the advantages of using the syslog-ng Store Box logserver appliance to collect, store, and manage system log (syslog) and eventlog messages for policy compliance.
Kontrolle und revisionssichere Auditierung privilegierter IT-ZugriffeBalaBit
In einem Unternehmen gibt es meist eine Vielzahl unbekannter privilegierter Benutzer: Systemadministratoren, Benutzer mit Zugriff auf vertrauliche Inhalte – besonders in IT-Architekturen, die mehrere Altsysteme enthalten.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
20240605 QFM017 Machine Intelligence Reading List May 2024
2015. Libre Software Meeting - syslog-ng: from log collection to processing and information extraction
1. syslog-ng: from log collection to
processing and information extraction
2015. Libre Software Meeting, Beauvais
Peter Czanik / BalaBit
2. 2
About me
■ Peter Czanik from Hungary
■ Community manager at BalaBit: syslog-ng upstream
■ Doing syslog-ng packaging, support, advocating
■ BalaBit is an IT security company with development HQ in Budapest,
Hungary
■ Over 200 employees: the majority are engineers
3. 4
Syslog → syslog-ng
■ Logging: recording events
■ Jan 14 11:38:48 linux-0jbu sshd[7716]: Accepted publickey for root from
127.0.0.1 port 48806 ssh2
■ syslog-ng: enhanced log daemon, with a focus on central log collection,
supporting a wide range of input and output methods with a flexible
configuration language
7. 8
syslog-ng: destinations
■ Traditional file and UDP/TCP/TLS destinations
■ SQL and NoSQL destinations (mysql, mongodb)
■ Visualization (graphite)
■ Alerting (riemann)
■ Message queuing (RabbitMQ, ZeroMQ)
■ Hadoop, Elasticsearch, Kafka and many more
8. 9
Configuration
■ “Don't Panic”
■ Simple and logical, even if looks difficult
■ Pipeline model:
□ Many different building blocks (sources, destinations, filters, parsers, etc.)
□ Connected using “log” statements into a pipeline
■ Sample config from Fedora
14. 15
Free-form log messages
■ Most log messages are: date + hostname + text
Mar 11 13:37:56 linux-6965 sshd[4547]: Accepted keyboard-
interactive/pam for root from 127.0.0.1 port 46048 ssh2
■ Text = English sentence with some variable parts
■ Easy to read by a human
15. 16
Why it does not scale
■ Information is presented differently by each application
■ Few logs (workstation) → easy to find information
■ Many logs (server) → difficult to find information
■ Difficult to process them with scripts
16. 17
Solution: structured logging
■ Events represented as name-value pairs
■ Example: an ssh login:
□ source_ip=192.168.123.45
□ app=sshd
□ user=root
■ Parsers in syslog-ng can turn unstructured and some structured data (csv,
JSON) into name value pairs
■ syslog-ng: name-value pairs inside
□ Date, facility, priority, program name, pid, etc.
■ Templates: use name-value pairs for custom file names or messages
19. 20
PatternDB parser
■ PatternDB message parser:
□ Can extract useful information from unstructured messages into name-value
pairs
□ Add status fields based on message text
□ Message classification (like LogCheck)
■ Needs XML describing log messages
■ Example: an ssh login failure:
□ user=root, source_ip=192.168.123.45, action=login, status=failure
□ classified as “violation”
21. 22
Creating patterns for syslog-ng: editor
■ Some sample patterns available:
□ https://github.com/balabit/syslog-ng-patterndb
■ Use an XML editor or text editor with syntax highlighting
■ Use “pdbtool” to
□ test, debug
□ merge
□ convert
patterns
22. 23
Creating patterns for syslog-ng: Puppet
■ More friendly format (especially if you use Puppet :-) )
■ https://github.com/ccin2p3/puppet-patterndb
■ Use “pdbtool” as usual
patterndb::simple::ruleset { 'myruleset':
id => '9586b525-826e-4c2d-b74f-381039cf470c',
patterns => [ 'sshd' ],
pubdate => '2014-03-24',
rules => [
{
id => 'd69bd1ed-17ff-4667-8ea4-087170cbceeb',
patterns => ['Successful login for user @QSTRING:user:"@ using method @QSTRING:method:"@']
}
]
}
23. 24
Creating patterns for syslog-ng: GUI
■ This is a work in progress
■ Finds patterns automagically from similar lines
■ Fields can be edited and named
■ Results can be verified
26. 27
Anonymizing messages
■ Many regulations about what can be logged
□ PCI-DSS: credit card numbers
□ Europe: IP addresses, user names
■ Locating sensitive information:
□ Regular expressions: slow, works also in unknown logs
□ Patterndb: fast, only in known log messages
■ Anonymizing:
□ Overwrite it with constant
□ Overwrite it with a hash of the original
27. 28
Language bindings in syslog-ng
■ The primary language of syslog-ng is C:
□ High performance: processes a lot more EPS than interpreted languages
■ Not everything is implemented in C
■ Rapid prototyping is easier in interpreted languages
■ Lua / Perl / Python / Java destinations, Lua monitoring source
□ Embedded interpreter
□ Message or full range of name value pairs can be passed
■ Java/Python moving from incubator to core in 3.7
28. 29
ElasticSearch through Java destination
■ Syslog-ng 3.7 beta has Java destination (originally in the incubator)
■ https://github.com/juhaszviktor/ESDestination
destination d_es {
java(
class_path("/usr/local/ESDestination.jar:/usr/share/elasticsearch/lib/*.jar")
class_name("org.syslog_ng.elasticsearch.ElasticSearchDestination")
option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}")
option("type", "test")
option("cluster", "syslog-ng")
option("flush_limit", "100")
option("custom_id", "$RCPTID")
);
};
29. 30
Central syslog-ng management
■ modules for Puppet, Salt and Ansible
■ Puppet is the most tested with thousands of machines
■ https://github.com/ihrwein/puppet-syslog_ng
■ Features:
□ Installs syslog-ng and sub-modules
□ Can configure syslog-ng with minimal limitations
30. 31
Questions? (and some answers)
■ Questions?
■ Some useful syslog-ng resources:
□ syslog-ng: http://syslog-ng.org/
□ ELSA (log analysis based on syslog-ng's patterndb):
http://code.google.com/p/enterprise-log-search-and-archive/
□ Alerting: http://devops.com/features/guide-modern-monitoring-alerting/
□ Mailing list: https://lists.balabit.hu/pipermail/syslog-ng/
□ My blog: http://czanik.blogs.balabit.com/
□ My e-mail: czanik@balabit.hu
34. 35
Interactive syslog-ng
■ See which path a log message takes inside syslog-ng
■ Stop at break points
■ Show current state of macros
■ Built-in help and tab completion
■ Initial commit in syslog-ng 3.7 (beta)
■ Feedback is very welcome!