This document contains multiple sections from Anton Chuvakin providing his top 11 reasons for various aspects of log management. The sections cover reasons to collect and preserve logs, look at logs, secure and protect logs, analyze logs, and finally a humorous take on reasons to hate logs. Anton is presented as a recognized security expert in log management and PCI compliance who has published extensively on these topics.
The presentation will describe methods for discovering interesting and actionable patterns in log files for security management without specifically knowing what you are looking for. This approach is different from "classic" log analysis and it allows gaining an insight into insider attacks and other advanced intrusions, which are extremely hard to discover with other methods. Specifically, I will demonstrate how data mining can be used as a source of ideas for designing future log analysis techniques, that will help uncover the coming threats. The important part of the presentation will be the demonstration how the above methods worked in a real-life environment.
What does "monitoring" mean? (FOSDEM 2017)Brian Brazil
Monitoring can mean very different things to different people, and this often leads to confusion and misunderstandings. There are many offerings both free software and commercials, and it's not always clear where each fits in the bigger picture. This talk will look a bit at the history of monitoring, and then into the general categories of Metrics, Logs, Profiling and Distributed tracing and how each of these is important in Cloud-based environment.
Video: https://www.youtube.com/watch?v=hCBGyLRJ1qo
Log Management 'Worst Practices' - log management tool from planning to deployment to operation. All the mistakes to avoid! All the pitfalls to skip! This was given at SANS Lunch and Learn a few times.
Using Logs for Breach Investigations and Incident Response by Dr Anton ChuvakinAnton Chuvakin
Log and logging overview
A brief on Incident response and forensics
Logs in incident investigations
Just what is log forensics?
Conclusions and call to action!
Become a Better Developer with Debugging Techniques for Drupal (and more!)Acquia
What is debugging? How is it different from simply writing a program, and how can you get better at it? A structured debugging approach narrows down problems, rather than using random changes and guesses, and can help you identify and solve problems faster and more effectively.
In this webinar about debugging techniques for Drupal, we’ll cover:
-A general approach to debugging Drupal problems
Common sources of bugs
-A tour of useful debugging tools and techniques that can help you start to see into the inner workings of any version of Drupal
-The use of tools such as XDebug, the devel suite, and client side debugging such as Firebug, LiveHTTPHeaders, and javascript debugging
Teach Your Sites to Call for Help: Automated Problem Reporting for Online Ser...Caktus Group
Keeping users happy after your site launches is easier if you know about problems before users complain. This presentation by Caktus developer Dan Poirier, delivered at PyTennessee 2018, lays out which problems you may want to be notified about and a number of tools that can help.
Metric Abuse: Frequently Misused Metrics in OracleSteve Karam
This is a presentation I created for RMOUG 2014 which I was sadly unable to attend. However, I wanted to share it with the Oracle community so that you can learn a bit about metrics that are frequently cited, frequently demonized, and frequently misused. In this deck we will go through the steps to diagnose issues and what NOT to blame as you go through the process.
The topics and concepts discussed here were originally formed in a blog post on the OracleAlchemist.com site: http://www.oraclealchemist.com/news/these-arent-the-metrics-youre-looking-for/
Talk given at Voices That Matter: Web Design in 2009. Although the examples are from web, it is equally (if not more) applicable to desktop, device, and mobile applications as well.
An idea for a log and backup policy that reduces the possibility of and potential damage from insider threats. Presented at Information Warfare Summit 2013.
Future of SOC: More Security, Less OperationsAnton Chuvakin
"Future of SOC: More Security, Less Operations" was originally presented by Dr Anton Chuvakin in March 2024 at a virtual conference in Finland
The future of SOC looks less like its past. AI is part of the future, but engineering-led approach to SOC is more critical
Detection and Response of the future will be more heavily automated
SOC Meets Cloud: What Breaks, What Changes, What to Do?Anton Chuvakin
SOC Meets Cloud: What Breaks, What Changes, What to Do?
originally presented at Mandiant mWise 2023 by Dr Anton Chuvakin of Google Cloud Office of the CISO
Cloud changes everything (does it though?), including how we do threat detection and incident response in the SOC. As we continue to transform our attack surfaces, how do we make sure our detection and response are done "the cloud way"? There were also cases where both business and IT migrated to the cloud, but security was left behind and had to approach cloud challenges with on-premise tools and practices. How should a SOC born before cloud deal with cloud? What to watch for? What changes? What breaks? What stays the same?
The presentation will describe methods for discovering interesting and actionable patterns in log files for security management without specifically knowing what you are looking for. This approach is different from "classic" log analysis and it allows gaining an insight into insider attacks and other advanced intrusions, which are extremely hard to discover with other methods. Specifically, I will demonstrate how data mining can be used as a source of ideas for designing future log analysis techniques, that will help uncover the coming threats. The important part of the presentation will be the demonstration how the above methods worked in a real-life environment.
What does "monitoring" mean? (FOSDEM 2017)Brian Brazil
Monitoring can mean very different things to different people, and this often leads to confusion and misunderstandings. There are many offerings both free software and commercials, and it's not always clear where each fits in the bigger picture. This talk will look a bit at the history of monitoring, and then into the general categories of Metrics, Logs, Profiling and Distributed tracing and how each of these is important in Cloud-based environment.
Video: https://www.youtube.com/watch?v=hCBGyLRJ1qo
Log Management 'Worst Practices' - log management tool from planning to deployment to operation. All the mistakes to avoid! All the pitfalls to skip! This was given at SANS Lunch and Learn a few times.
Using Logs for Breach Investigations and Incident Response by Dr Anton ChuvakinAnton Chuvakin
Log and logging overview
A brief on Incident response and forensics
Logs in incident investigations
Just what is log forensics?
Conclusions and call to action!
Become a Better Developer with Debugging Techniques for Drupal (and more!)Acquia
What is debugging? How is it different from simply writing a program, and how can you get better at it? A structured debugging approach narrows down problems, rather than using random changes and guesses, and can help you identify and solve problems faster and more effectively.
In this webinar about debugging techniques for Drupal, we’ll cover:
-A general approach to debugging Drupal problems
Common sources of bugs
-A tour of useful debugging tools and techniques that can help you start to see into the inner workings of any version of Drupal
-The use of tools such as XDebug, the devel suite, and client side debugging such as Firebug, LiveHTTPHeaders, and javascript debugging
Teach Your Sites to Call for Help: Automated Problem Reporting for Online Ser...Caktus Group
Keeping users happy after your site launches is easier if you know about problems before users complain. This presentation by Caktus developer Dan Poirier, delivered at PyTennessee 2018, lays out which problems you may want to be notified about and a number of tools that can help.
Metric Abuse: Frequently Misused Metrics in OracleSteve Karam
This is a presentation I created for RMOUG 2014 which I was sadly unable to attend. However, I wanted to share it with the Oracle community so that you can learn a bit about metrics that are frequently cited, frequently demonized, and frequently misused. In this deck we will go through the steps to diagnose issues and what NOT to blame as you go through the process.
The topics and concepts discussed here were originally formed in a blog post on the OracleAlchemist.com site: http://www.oraclealchemist.com/news/these-arent-the-metrics-youre-looking-for/
Talk given at Voices That Matter: Web Design in 2009. Although the examples are from web, it is equally (if not more) applicable to desktop, device, and mobile applications as well.
An idea for a log and backup policy that reduces the possibility of and potential damage from insider threats. Presented at Information Warfare Summit 2013.
Future of SOC: More Security, Less OperationsAnton Chuvakin
"Future of SOC: More Security, Less Operations" was originally presented by Dr Anton Chuvakin in March 2024 at a virtual conference in Finland
The future of SOC looks less like its past. AI is part of the future, but engineering-led approach to SOC is more critical
Detection and Response of the future will be more heavily automated
SOC Meets Cloud: What Breaks, What Changes, What to Do?Anton Chuvakin
SOC Meets Cloud: What Breaks, What Changes, What to Do?
originally presented at Mandiant mWise 2023 by Dr Anton Chuvakin of Google Cloud Office of the CISO
Cloud changes everything (does it though?), including how we do threat detection and incident response in the SOC. As we continue to transform our attack surfaces, how do we make sure our detection and response are done "the cloud way"? There were also cases where both business and IT migrated to the cloud, but security was left behind and had to approach cloud challenges with on-premise tools and practices. How should a SOC born before cloud deal with cloud? What to watch for? What changes? What breaks? What stays the same?
Meet the Ghost of SecOps Future by Anton ChuvakinAnton Chuvakin
Meet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future
Today’s SOC has an increasingly difficult job protecting growing and expanding organizations. The landscape is changing and the SOC needs to change with the times or risk falling behind the evolution of business, IT, and threats.
But you have choices! Your future fate is not set in stone and can be changed: some optimize what they have without drastic upheaval, while others choose to truly transform their detection and response.
Join us as we show you a vision of what the SOC will look like in the near future and how to choose the best course of action today.
Originally aired at https://cloudonair.withgoogle.com/events/2023-dec-security-talks
Video https://youtu.be/KbQbuFAPY2c?si=0llv1v_CkVtvsyms
SOC Lessons from DevOps and SRE by Anton ChuvakinAnton Chuvakin
SOC Lessons from DevOps and SRE by Dr Anton Chuvakin - RSA 2023 Google Cloud sideshow presentation focused on using select DevOps and SRE lessons to make your SOC better
20 years of SIEM was prepared for the SANS webinar https://www.sans.org/webcasts/anton-chuvakin-discusses-20-years-of-siem-what-s-next/ and offers Anton's reflection on SIEM past and future
10X SOC - SANS Blue Summit Keynote 2021 - Anton ChuvakinAnton Chuvakin
Can We REALLY 10X the SOC? by Dr Anton Chuvakin
Many organizations promise to transform your security operations center (SOC) with technology, advice or their personnel. However, what does it take to really transform your SOC to be ready for future threats? Is this an impossible problem? Is this something that can be only done by well funded organizations? Let's explore these and other questions in this talk.
https://www.sans.org/cyber-security-training-events/blue-team-summit-2021/#agenda
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin
End-User Case Study: Five Best and Five Worst Practices for SIEM
Implementing SIEM sounds straightforward, but reality sometimes begs to differ. In this session, Dr.
Anton Chuvakin will share the five best and worst practices for implementing SIEM as part of security
monitoring and intelligence. Understanding how to avoid pitfalls and create a successful SIEM
implementation will help maximize security and compliance value, and avoid costly obstacles,
inefficiencies, and risks
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin
End-User Case Study: Five Best and Five Worst Practices for SIEM
Implementing SIEM sounds straightforward, but reality sometimes begs to differ. In this session, Dr.
Anton Chuvakin will share the five best and worst practices for implementing SIEM as part of security
monitoring and intelligence. Understanding how to avoid pitfalls and create a successful SIEM
implementation will help maximize security and compliance value, and avoid costly obstacles,
inefficiencies, and risks
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
1. Originally posted on my blog “Security Warrior” www.securitywarrior.org –
reposted here all in one place.
DISCLAIMER:
Security is a rapidly changing field of human endeavor. Threats we face literally
change every day; moreover, many security professionals consider the rate of
change to be accelerating. On top of that, to be able to stay in touch with such
ever-changing reality, one has to evolve with the space as well. Thus, even
though I hope that this document will be useful for to my readers, please keep in
mind that is was possibly written years ago. Also, keep in mind that some of the
URL might have gone 404, please Google around.
Anton's "Top 11 Reasons to Collect and Preserve Computer Logs", presented in no
particular order:
1. Before anything else, do you deal with credit cards? Patient info? Are you a
government org under FISMA? A financial org? You have to keep'em - stop
reading further.
2. What if there is a law or a regulation that requires you to retain logs - and you
don't know about it yet? Does the world "compliance" ring a bell?
3. An auditor comes and asks for logs. Do you want to respond "Eh, what do you
mean?"?
4. A system starts crashing and keeps doing so. Where is the answer? Oops, it was in
the logs - you just didn't retain them ...
5. Somebody posts a piece of your future quarterly report online. Did John Smith did
it? How? If not him, who did? Let's see who touched this document, got logs?
6. A malware is rampant on your network. Where it came from? Who spreads it?
Just check the logs - but only if you have them saved.
7. Your boss comes and says 'I emailed you this and you ignored it!!' - 'No, you
didn't!!!' Who is right? Only email logs can tell!
8. Network is slow; somebody is hogging the bandwidth. Let's catch the bastard! Is
your firewall logging? Keep the info at least until you can investigate.
9. Somebody added a table to your database. Maybe he did something else too - no
change control forms were filed. Got database log management? How else would
you know?
10. Disk space is cheap; tape is cheaper still. Save a log! Got SAN or NAS? Save a
few of them!
11. If you plan to throw away a log record, think - are you 100% sure you won't need
it, ever? Exactly! :-) Keep it.
Anton's "Top 11 Reasons to Look at Your Logs", presented in no particular order:
2. 1. The first reason is again disarmingly simple (is it :-)). Read PCI DSS lately?
Glanced at HIPAA? Suffer under FISMA? Yup, all of the above say that you
must not only have, but also review logs periodically.
2. Are you 0wned? How do you know if all your logs are stashed on a tape in a
closet? Look at them! Now!!
3. An incident happens. Really, who needs extra motivation to look at logs in this
case? Duh! Logs for incident response is a "no-brainer" use case for log review.
4. Users - from CEO to a janitor. You might have to know what they do on your IT
systems! How? Read the logs! Everybody leaves tracks.
5. Logged system errors. Sometimes they are stupid, sometimes - benign. However,
often they mean that "stuff" is about to hit the fan. Periodic review of logs reveals
them and saves the day.
6. Network slowed to a crawl? Applications are slooow? Server is not ... well,
serving? :-) Where is the answer? In the logs, but you need to read them and
understand them.
7. That policy you wrote a few months ago. Anybody following that? Anybody
remembers that? Halloooo! Check the logs and you'd know.
8. You know your auditor might check your logs. But did you know they might also
check whether you looked at them? Did'ya? Review the logs and leave the record
of this activity!
9. Change can be good. But then again, it may be the sign that your controls are
lacking. Who changes what and when? From what and to what? Just review the
logs.
10. Now, you hate looking at logs. You have too many! In this case, look at a specific
subset of logs that you never saw before- NBS. Or just deploy log
management that can do it for you.
11. Logs can help you predict the future (if you review, know and love them :-)).
Don't believe it? If you read them for long enough, you develop an ability to -
gasp!- predict the future, albeit mostly future problems :-)
Anton’s "Top 11 Reasons to Secure and Protect Your Logs", presented in no
particular order:
1. Let's review why you are reviewing logs. Will logs that might have been changed
by somebody, somewhere, somehow still be useful for items 1-11 from here? No?
Secure them!
2. Oooh, logs in court? Challenges abound! To respond to them, one needs to protect
the logs so you can claim that they are both authentic and reliable.
3. A human error still beats an evil hacker as the main cause of IT problems. Are
your logs safe from it? Available when needed? Protect them from crashes and
other faults!
3. 4. PCI DSS just says so: "Secure audit trails so they cannot be altered." Wonna do it-
or pay the fines?
5. Do you protect financial records? Identity info? Passwords? Some of it ends up in
logs - thus making them more sensitive. Secure the C-I-A of logs!
6. Do you look at logs during incident investigation? Do you want them to be "true"
or full of random (if creative...) cr*p, inserted by the guilty party? Secure the logs!
7. Think that "attacks vs logging" are theoretical? Think again. Are your logs safe or
vulnerable? Is your logging tool 0wned?
8. Syslog + UDP = log injection. Are you protected (reliable TCP, confirmed
delivery, encryption - SSH, SSL, VPN)?
9. Why change logs? No, really, why change logs? If you never change logs - and
you never should - hash them right away after collection to make them
immutable.
10. Logs are backed up on tape - who will see them? Well, whoever restores the tape,
that's who! Encrypt them to protect them from accidental and malicious disclosure
if tape is lost.
11. Why log access to logs? Same reason why you had the logs in the first place - to
review who did what. Who broke through and stole the logs? Who browsed them
without permission? Only logs will tell - if you have them!
Anton’s "Top 11 Reasons to Analyze Your Logs" Don't just read your logs; analyze
them. Why? Here are the reasons:
1. Seen an obscure log message lately? Me too - in fact, everybody have. How do
you know what it means (and logs usually do mean something) without analysis?
At the very least, you need to bring additional context to know what some logs
mean.
2. Logs often measure in gigabytes and soon will in terabytes; log volume grows all
the time - it passed a limit of what a human can read a long time ago, it then made
simple filtering 'what logs to read' impossible as well: automated log analysis is
the only choice.
3. Do you peruse your logs in real time? This is simply absurd! However, automated
real-time analysis is entirely possible (and some logs do crave for your attention
ASAP!)
4. Can you read multiple logs at the same time? Yes, kind of, if you print them out
on multiple pages to correlate (yes, I've seen this done :-)). Is this efficient? God,
no! Correlation across logs of different types is one of the most useful approaches
to log analysis.
5. A lot of insight hides in "sparse" logs, logs where one record rarely matters, but a
large aggregate does (e.g. from one "connection allowed" firewall log to a scan
pattern). Thus, the only way to extract that insight from a pool of data is through
algorithms (or, as some say, visualization)
6. Ever did a manual log baselining? This is where you read the logs and learn
which ones are normal for your environment. Wonna do it again? :-) Log baseline
4. learning is a useful and simple log analysis technique, but humans can only do it
for so much.
7. OK, let's pick the important logs to review. Which one are those? The right
answer is "we don't know, until we see them." Thus, to even figure out which logs
to read, you need automated analysis.
8. Log analysis for compliance? Why, yes! Compliance is NOT only about log
storage (e.g. see PCI DSS). How to highlight compliance-relevant messages?
How to see which messages will lead to a violation? How do you satisfy those
"daily log review" requirements? Thru automated analysis, of course!
9. Logs allow you to profile your users, your data and your resources/assets.
Really? Yes, really: such profiling can then tell you if those users behave in an
unusual manner (in fact, the oldest log analysis systems worked like that). Such
techniques may help reach the holy grail of log analysis: automatically tell you
what matters for you!
10. Ever tried to hire a log analysis expert? Those are few and far between. What if
your junior analysts can suddenly analyze logs just as well? One log analysis
system creator told me that his log data mining system enabled exactly that. Thus,
saving a lot of money to his organization.
11. Finally, can you predict future with your logs? I hope so! Research on predictive
analytics is ongoing, but you can only do it with automated analysis tools, not
with just your head alone (no matter how big :-)) ...
Finally, HUMOROUS (posted on April 1st)
Anton’s Top 11 Reasons to Hate Logs
You thought I am done with my Top 11 lists? Nah... here is one more, which
actually is designed to bite you in the ass on a certain date. So, "Top 11
Reasons to HATE Logs ... With a Passion."
1. Read any logs lately? Got bored in 5 minutes - or survived for the
whopping 10? Congrats, you score a point! But logs are
still boooooooooooooooooooooooooooooring.
2. One log, two logs, 10 logs.... 1,000,000,000 logs: rabbits and
hamsters cannot match the speed with whichlogs multiply. Don't you
just hate that?
3. You keep hearing people refer to "log data." Then you run 'tail
/var/log/messages' and see text in pidgin English. Where is my data?
Hate it!
4. "Real hackers don't get logged": thus logs are seen as useless - and
hated by some "hard core" security pros!
5. 5. If people lie to you, you hate it. Logs do lie too (see 'false positives')
- and they are hated too.
6. 'Transport error 202 message repeated 3456 times.' Niiiiice. Now go
fix that! Fix what? Ah, hate the log obscurity!
7. Why are there 47 different ways to log that "connection from A to B
was established OK?" Or 21 way to say "user logged in OK?" No,
really? Why? Who can I kill to stop this insanity?
8. You MUST do XYZ with logs for compliance. Or you are going to jail,
buddy! No, sorry, we can't tell you what XYZ is. Maybe in 7 years; for
now, just store everything.
9. 'Critical error: process completed successfully' and
'Operation successfully failed' engender deep and lasting hatred of
logs in most people. They just do ...
10. The book called "Ugliest Logs Ever!" is a fat tome, covering every log
source from a Linux system all the way to databases and CRM. Bad
logs are popular! Bad logs are all the rage among the programmers!
Bad logs are here to stay. Bad logs that mean nothing power the log
hatred.
11. "Logs: can't live with them, can't live without them" :-) Hate them we
might for different reasons, but we still must collect, protect, review,
and analyze them ...
ABOUT AUTHOR:
This is an updated author bio, added to the paper at the time of reposting in
2009.
Dr. Anton Chuvakin (http://www.chuvakin.org) is a recognized security expert in
the field of log management and PCI DSS compliance. He is an author of books
"Security Warrior" and "PCI Compliance" and a contributor to "Know Your Enemy
II", "Information Security Management Handbook" and others. Anton has
published dozens of papers on log management, correlation, data analysis, PCI
DSS, security management (see list www.info-secure.org) . His blog
http://www.securitywarrior.org is one of the most popular in the industry.
In addition, Anton teaches classes and presents at many security conferences
across the world; he recently addressed audiences in United States, UK,
Singapore, Spain, Russia and other countries. He works on emerging security
standards and serves on the advisory boards of several security start-ups.
Currently, Anton is developing his security consulting practice, focusing on
logging and PCI DSS compliance for security vendors and Fortune 500
6. organizations. Dr. Anton Chuvakin was formerly a Director of PCI Compliance
Solutions at Qualys. Previously, Anton worked at LogLogic as a Chief Logging
Evangelist, tasked with educating the world about the importance of logging for
security, compliance and operations. Before LogLogic, Anton was employed by a
security vendor in a strategic product management role. Anton earned his Ph.D.
degree from Stony Brook University.