“ Grand Challenges” of Log Management Dr Anton Chuvakin Chief Logging Evangelist LogLogic, Inc Mitigating Risk. Automating Compliance.

Who is Anton? Dr. Anton Chuvakin from LogLogic “is probably the number one authority on system logging in the world ” SANS Institute (2008) ( http://www.sans.edu/resources/securitylab/loglogic_chuvakin.php )

Dr. Anton Chuvakin from LogLogic “is probably the number one authority on system logging in the world ”

SANS Institute (2008)

( http://www.sans.edu/resources/securitylab/loglogic_chuvakin.php )

Outline Log Management Intro Innovation: BIG vs Small Step away from small, tactical “issues” for a second … “ Grand Challenges” of Log Management How you can help?!

Log Management Intro

Innovation: BIG vs Small

Step away from small, tactical “issues” for a second …

“ Grand Challenges” of Log Management

How you can help?!

Why “Grand Challenges”? Log management BIG and unsolved problems that cause major pain! Problems that people tried to solve – and FAILED! From collection to decision-making based on logs; from compliance to security and operations; from today’s log sources to the future – there are challenges everywhere!

Log management BIG and unsolved problems that cause major pain!

Problems that people tried to solve – and FAILED!

From collection to decision-making based on logs; from compliance to security and operations; from today’s log sources to the future – there are challenges everywhere!

GC1 – Secure and Reliable Log Collection Challenge To collect the logs securely, reliably AND without heavy management overhead and complexity of access Why a grand challenge? Agents vs remote grabbing vs stream: all suck. Security and reliability cost major management overhead Current approaches? Agents + remote grab (administrator access) + stream (syslog) Why still a challenge? All approaches have critical drawbacks

Challenge

To collect the logs securely, reliably AND without heavy management overhead and complexity of access

Why a grand challenge?

Agents vs remote grabbing vs stream: all suck. Security and reliability cost major management overhead

Current approaches?

Agents + remote grab (administrator access) + stream (syslog)

Why still a challenge?

All approaches have critical drawbacks

GC2 - Log Parsing and Regexs Challenge To turn logs into information, one needs to parse them; to parse them one needs [typically] expert-created regular expressions (regex’s) Why a grand challenge? Every log type requires hand-writing a set of regexes Current approaches? UIs, “semi-auto”/assisted regex creators, limited auto-extraction, choosing not to parse, etc Why still a challenge? Despite all tools, log expert must create the rules

Challenge

To turn logs into information, one needs to parse them; to parse them one needs [typically] expert-created regular expressions (regex’s)

Why a grand challenge?

Every log type requires hand-writing a set of regexes

Current approaches?

UIs, “semi-auto”/assisted regex creators, limited auto-extraction, choosing not to parse, etc

Why still a challenge?

Despite all tools, log expert must create the rules

GC3 – Fast Ad Hoc Summarization Challenge Everybody wants reports on this and that NOW! They rarely know what is ‘this’ and ‘that’ Why a grand challenge? Its either fast or ad hoc, not both. Users want both! Current approaches? Database tuning; custom indices; “non-RDBMS with a little bit of RDBMS” Why still a challenge? None really work or require expert tuning; pick lesser evil

Challenge

Everybody wants reports on this and that NOW! They rarely know what is ‘this’ and ‘that’

Why a grand challenge?

Its either fast or ad hoc, not both. Users want both!

Current approaches?

Database tuning; custom indices; “non-RDBMS with a little bit of RDBMS”

Why still a challenge?

None really work or require expert tuning; pick lesser evil

GC4 – Automated Meaning Extraction Challenge Automatically analyze logs and gain useful information, across domains (security, ops, compliance) Why a grand challenge? Log analysis is heavily manual, interpretative and domain- and system-specific Current approaches? Rule-based, summarization, filtering, minimum anomaly detection Why still a challenge? “ Log analysis is an art, not science” -> no automation

Challenge

Automatically analyze logs and gain useful information, across domains (security, ops, compliance)

Why a grand challenge?

Log analysis is heavily manual, interpretative and domain- and system-specific

Current approaches?

Rule-based, summarization, filtering, minimum anomaly detection

Why still a challenge?

“ Log analysis is an art, not science” -> no automation

GC5 – Scalable Data Presentation Challenge How to present log massive volumes of log data to users to help them solve their problems across domains Why a grand challenge? Tables, pie charts, graphs all leave much to be desired; lose information and don’t scale Current approaches? Table, pie/bar chart, graphs, “advanced” visualization Why still a challenge? No effective method is invented yet

Challenge

How to present log massive volumes of log data to users to help them solve their problems across domains

Why a grand challenge?

Tables, pie charts, graphs all leave much to be desired; lose information and don’t scale

Current approaches?

Table, pie/bar chart, graphs, “advanced” visualization

Why still a challenge?

No effective method is invented yet

GC6 – “Fuzzy” Search Challenge How to find the “right” log message (s) without knowing what to look for, exactly? Why a grand challenge? Many uses of logs require searching but users often don’t know what to look for Current approaches? Trying keywords + wildcards + refining search as we go Why still a challenge? No method to incorporate uncertainty in search is found yet

Challenge

How to find the “right” log message (s) without knowing what to look for, exactly?

Why a grand challenge?

Many uses of logs require searching but users often don’t know what to look for

Current approaches?

Trying keywords + wildcards + refining search as we go

Why still a challenge?

No method to incorporate uncertainty in search is found yet

Call to Action! Simple  Pick a challenge and solve it!!! Come to discuss!! Act on ideas! Send ideas! Explore!

Simple 

Pick a challenge and solve it!!!

Come to discuss!!

Act on ideas!

Send ideas!

Explore!

Thank You! Anton Chuvakin, Ph.D. www.chuvakin.org Chief Logging Evangelist LogLogic, Inc www.loglogic.com See www.info-secure.org for my papers, books, reviews and other security and logging resources. Subscribe to my blog at www.securitywarrior.org

Anton Chuvakin, Ph.D. www.chuvakin.org

Chief Logging Evangelist

LogLogic, Inc www.loglogic.com

See www.info-secure.org for my papers, books, reviews

and other security and logging resources.

Subscribe to my blog at www.securitywarrior.org

Further Reading / Blog Posts “Idea Log Management Tool” ( http://chuvakin.blogspot.com/2007/11/ideal-log-management-tool.html ) “Future Problems -1” ( http://chuvakin.blogspot.com/2008/06/ideal-tool-to-solve-real-problems-of.html ) “Future Problems -2” ( http://chuvakin.blogspot.com/2008/08/ideal-tool-to-solve-real-problems-of.html )

“Idea Log Management Tool” ( http://chuvakin.blogspot.com/2007/11/ideal-log-management-tool.html )

“Future Problems -1” ( http://chuvakin.blogspot.com/2008/06/ideal-tool-to-solve-real-problems-of.html )

“Future Problems -2” ( http://chuvakin.blogspot.com/2008/08/ideal-tool-to-solve-real-problems-of.html )

Other Candidate Challenges

GCC1 – “Bad” Logs Challenge Many logs just don’t have the right information in them; correlated logs; logs w missing info; multi-file logs, etc Why a grand challenge? Making sense of logs is hard if key information is missing Current approaches? Ignore the problem, try to manually enrich the information from other sources Why still a challenge? The above doesn’t help; deeper change in how logging is done is probably needed

Challenge

Many logs just don’t have the right information in them; correlated logs; logs w missing info; multi-file logs, etc

Why a grand challenge?

Making sense of logs is hard if key information is missing

Current approaches?

Ignore the problem, try to manually enrich the information from other sources

Why still a challenge?

The above doesn’t help; deeper change in how logging is done is probably needed

GCC2 – Log Chaos Challenge Logs come in a dizzying variety of formats, via different ports, they look different – how do we understand them? Why a grand challenge? Lack of log standards make log analysis unreliable and complex art Current approaches? Take logs one by one; write regexes or index Why still a challenge? No log standard has been created yet; CEE is working on it! ( http://cee.mitre.org )

Challenge

Logs come in a dizzying variety of formats, via different ports, they look different – how do we understand them?

Why a grand challenge?

Lack of log standards make log analysis unreliable and complex art

Current approaches?

Take logs one by one; write regexes or index

Why still a challenge?

No log standard has been created yet; CEE is working on it! ( http://cee.mitre.org )

GCC3 – Unified Log Storage Data Model Challenge Logs come in a dizzying variety of formats, some can be parsed, some can’t – how to store them for quick and smart access (not “slow and painful”) Why a grand challenge? Logs are just too different Current approaches? RDBMS (one vs many) vs flat files vs custom vs … Why still a challenge? Despite the effort, many limitations persists

Challenge

Logs come in a dizzying variety of formats, some can be parsed, some can’t – how to store them for quick and smart access (not “slow and painful”)

Why a grand challenge?

Logs are just too different

Current approaches?

RDBMS (one vs many) vs flat files vs custom vs …

Why still a challenge?

Despite the effort, many limitations persists