Welcome Webinar
John Merideth
Technical Trainer
jmerideth@sumologic.com
Sumo Logic confidential
Course Agenda
Data collection
Search and analyze
15 min.
15 min.
Visualize and monitor
15 min.
Q & A
Overview
5 min.
15 min.
Sumo Logic confidential
Our analytics
Your data
Sumo Logic Continuous Intelligence PlatformTM
Operational Intelligence Security Intelligence Business Intelligence Global Intelligence
Your people DevOps Engineering IT Ops Customer Product Data
SecOps
Success Scientist
Apps MicroServices Cloud Systems & SaaS Service Mobile devices 3rd
Party
infrastructure
systems & feeds
Multi-tenant Scalable & Machine API Advanced
Security & elastic learning analytics
monitoring
Sumo Logic confidential
Sumo Logic Data Flow
Alerts
Dashboards
3
Visualize & Monitor
Operators
Charts
2
Search & Analyze
Collectors
Sources
1
Data Collection
Sumo Logic confidential
Login to training environment
url: service.sumologic.com
email:
training+analyst###@sumologic.com
password: Security21!
### - a number between 001-999, for
example
training+analyst057@sumologic.com
Note: Place your ### number into chat so
that everyone knows not to use the one
you selected
7
Collectors
Sources
1
Data Collection
Data Collection Activity 1
1. In the left navigation pane click Manage Data> Collection
> Collection
2. In the magnifying glass enter okta and press return
3. Notice the two collectors and the sources of data below
them
Sumo Logic confidential
Host Metrics
Logs-to-
Metrics
Docker Stats
AWS Metrics
+ Metadata
Streaming
Metrics
HTTP
Metrics
Sumo Logic confidential
Sends the data to the
Sumo service
Encrypts and
compresses the data
Installed Collector Overview
Collects logs and metrics
from its Sources
Installed Collector
Java Agent
Sumo Logic confidential
Sending Data ⇨ Metadata
Metadata tags are associated with each log message that is collected.
Tag Description
_collector Name of the collector (defaults to hostname)
_sourceHost Hostname of the server (defaults to hostname)
_sourceName Name and Path of the log file
_source Name of the source this data came through
_sourceCategory
Can be freely configured. Main metadata tag
(e.g. labs/apache/access)
Sumo Logic confidential
Metadata: Source Category Best Practices and Benefits
Common components (and any combination of):
• Environment (Prod/UAT/DEV)
• Application Name
• Geographic Information (East vs West datacenter, office location, etc.)
• AWS Region
• Business Unit
Highest level components should group the data how it is most often search together:
Prod/Web/Apache/Access
Dev/Web/Apache/Access
Prod/DB/MySQL/Error
Dev/DB/MySQL/Error
Web/Apache/Access/Prod
Web/Apache/Access/Dev
DB/MySQL/Error/Prod
DB/MySQL/Error/Dev
Sumo Logic confidential
What Data can I Analyze?
Option 1
Explore your Collectors
Option 2
Search for source categories
Sumo Logic confidential
Operators
Charts
2
Search & Analyze
Search & Analyze Activity 2
1. To open a query, at the top, click +New and select Log
Search
2. At the top query builder paste the following code (I will put
in chat for you to copy)
_sourceCategory=Labs/Apache/Access and "Mozilla"
| parse ""GET *" * " as url,status_code
| where status_code matches "5*"
| count by status_code
| sort by _count
1. You can modify the time to Last 60 minutes and click
Start
2. You will have a table displayed and you can click on other
chart types
Sumo Logic confidential
Data Analytics ⇨ Query Syntax
Syntax: metadata
Keywords and operators, separated by pipes, that build on top of each other
parse
filter
aggregate
format
keywords
_sourceCategory=Labs/Apache/Access and "Mozilla"
| parse ""GET *" * " as url,status_code
| where status_code matches “5*”
| count by status_code
| sort by _count
| limit 3
Sumo Logic confidential
Data Analytics ⇨ Query Syntax
Syntax:
Use metadata and keywords to narrow your search scope
Results
keyword
metadata keyword
| parse
| filter
| aggregate
| format
metadata + keywords
_sourceCategory=Labs/Apache/Access and "Mozilla"
Sumo Logic confidential
Data Analytics ⇨ Query Syntax
Syntax:
Extract meaningful fields to provide structure to your data
Parse Anchor:
| parse " *@* " as user,domain
Parse Regex:
| parse regex "(?<src_ip>d{1,3}
.d{1,3}.d{1,3}.d{1,3})”
Other Parse Operators:
csv, json, keyvalue, split, xml
Learn more: Parse Operators
| parse
| filter
| aggregate
| format
metadata + keywords
Sumo Logic confidential
Data Analytics ⇨ Query Syntax
Syntax:
Further filter results using your extracted fields
where operator:
| where !(status_code=304)
in operator:
| if(status_code in("501","502"),
"Error","OK") as code_type
Other Filter Operators:
join, lookup, matches, filter,
isEmpty, isNull, isBlank
Learn more: Filter operator example
| parse
| filter
| aggregate
| format
metadata + keywords
Sumo Logic confidential
Data Analytics ⇨ Query Syntax
Syntax:
Evaluate messages and place them into groups
count operator:
| count by status_code
avg operator:
| avg(size) by src_ip
pct operator:
| pct(filesize,75) by _sourceHost
Other Aggregation Operators:
sum, count_distinct, stddev, min,
max
Learn more: Aggregation operators
| parse
| filter
| aggregate
| format
metadata + keywords
Sumo Logic confidential
Data Analytics ⇨ Query Syntax
Syntax:
Format to display desired results succinctly
top operator:
| top 5 src_ip by avg_size
fields operator:
| fields src_ip, avg_size
transpose operator:
| transpose row src_ip column url
Other formatting Operators:
format, formatdate, limit, sort
Learn more: Trends over time using transpose
| parse
| filter
| aggregate
| format
metadata + keywords
Sumo Logic confidential
Advanced Analytics
Geo Lookup
_sourceCategory=Labs/Apache/Access
| lookup latitude, longitude from geo://default on ip=src_ip
| count by latitude, longitude
Outlier
_sourceCategory=Labs/Apache/Access and status_code=404
| timeslice 1m
| count(status_code) as error_count by _timeslice
| outlier error_count
Predict
_sourceCategory=Labs/Apache/Access
| timeslice 5m
| count as requests by _timeslice
| predict requests by 5m forecast=12
Log operators Cheat Sheet: https://help.sumologic.com/05Search/Search-Cheat-
Sheets/Log-Operators-Cheat-Sheet
Sumo Logic confidential
Advanced Analytics
Find the “needle in the haystack” by identifying patterns.
Compare today’s patterns with patterns in the past.
_sourceCategory=Labs/snort
| logreduce
_sourceCategory=Labs/snort
| logcompare timeshift -24h
LogReduce
LogCompare
Sumo Logic confidential
Alerts
Dashboards
3
Visualize & Monitor
Visualize & Monitor Activity 3
1. To create a dashboard, in the middle right click Add to
Dashboard
2. In the popup window under Panel Title enter Track 500s
under Dashboard enter Apache_db_<your initials###>
1. At the bottom of the popup click Add
1. To create an alert, in the left navigation pane click Manage
Data> Alert
2. In the upper right corner click Add and select New
Monitor
Sumo Logic confidential
Monitoring - Dashboards
• Each Panel processes results from
a single search
• Drill down into corresponding
query or link to another Dashboard
• Live Mode: provides live stream of
data
• Use Dashboards as templates with
Filters
Sumo Logic confidential
Sumo Logic Data Flow
Alerts
Dashboards
3
Visualize & Monitor
Operators
Charts
2
Search & Analyze
Collectors
Sources
1
Data Collection
What else may we address for you?
Sumo Logic confidential
• Onboarding Checklist
https://help.sumologic.com/01Sta
rt-Here/Onboarding_Checklist
• Take the training
https://www.sumologic.com/learn
/training/
• Read the docs
https://help/sumologic.com
Want to learn more?
Sumo Logic Confidential
Thank you
Sumo Logic Confidential
Sumo Logic Confidential
Which Topic would you like covered next?
(Single Choice)
Answer 1: Logreduce Operator
Answer 2: New Dashboards
Answer 3: Best Practices for Search Queries
Answer 4: Partitions
Answer 5: Collector Setup Optimization
Sumo Logic confidential
LogReduce – Foundation for Anomaly Detection
Reduce hundreds of thousands of pages of results into a single page of
meaningful patterns.
Under the Covers:
• LogReduce deconstructs log messages into their most basic patterns to
facilitate overall behavioral analysis – at the printf level
• Log messages are converted into unique hashed signatures – these
signatures are the building blocks of anomaly detection
– Events that occur more than others (e.g. errors flooding your
logs)
– Events that occur very infrequently but are important (e.g. rare
exception)
• Edit each signature to tailor each experience
• Benefit from machine learning that improves over time based on your
data and activity
Sumo Logic confidential
New Dashboards
• New charts, like Honeycomb
• Full control over look and feel with
JSON
• Build panels directly in the
dashboard
• Advanced filtering and metrics
query building
• Basic charts, like time series and
categorical
• Few color and font choices
• Panels created from search and
metrics tabs
• Limited filters and queries
• Still supported
Classic Dashboard New Dashboard
About New dashboards: https://help.sumologic.com/Visualizations-and-
Alerts/Dashboard_(New)/About_Dashboard_(New)
Sumo Logic confidential
Best Practices for Search Queries
Sumo Logic confidential
Partitions
Sumo Logic confidential
Collector Setup Optimization
Sumo Logic Confidential
Thank you

Welcome Webinar Slides

  • 1.
    Welcome Webinar John Merideth TechnicalTrainer jmerideth@sumologic.com
  • 2.
    Sumo Logic confidential CourseAgenda Data collection Search and analyze 15 min. 15 min. Visualize and monitor 15 min. Q & A Overview 5 min. 15 min.
  • 3.
    Sumo Logic confidential Ouranalytics Your data Sumo Logic Continuous Intelligence PlatformTM Operational Intelligence Security Intelligence Business Intelligence Global Intelligence Your people DevOps Engineering IT Ops Customer Product Data SecOps Success Scientist Apps MicroServices Cloud Systems & SaaS Service Mobile devices 3rd Party infrastructure systems & feeds Multi-tenant Scalable & Machine API Advanced Security & elastic learning analytics monitoring
  • 4.
    Sumo Logic confidential SumoLogic Data Flow Alerts Dashboards 3 Visualize & Monitor Operators Charts 2 Search & Analyze Collectors Sources 1 Data Collection
  • 5.
    Sumo Logic confidential Loginto training environment url: service.sumologic.com email: training+analyst###@sumologic.com password: Security21! ### - a number between 001-999, for example training+analyst057@sumologic.com Note: Place your ### number into chat so that everyone knows not to use the one you selected
  • 6.
    7 Collectors Sources 1 Data Collection Data CollectionActivity 1 1. In the left navigation pane click Manage Data> Collection > Collection 2. In the magnifying glass enter okta and press return 3. Notice the two collectors and the sources of data below them
  • 7.
    Sumo Logic confidential HostMetrics Logs-to- Metrics Docker Stats AWS Metrics + Metadata Streaming Metrics HTTP Metrics
  • 8.
    Sumo Logic confidential Sendsthe data to the Sumo service Encrypts and compresses the data Installed Collector Overview Collects logs and metrics from its Sources Installed Collector Java Agent
  • 9.
    Sumo Logic confidential SendingData ⇨ Metadata Metadata tags are associated with each log message that is collected. Tag Description _collector Name of the collector (defaults to hostname) _sourceHost Hostname of the server (defaults to hostname) _sourceName Name and Path of the log file _source Name of the source this data came through _sourceCategory Can be freely configured. Main metadata tag (e.g. labs/apache/access)
  • 10.
    Sumo Logic confidential Metadata:Source Category Best Practices and Benefits Common components (and any combination of): • Environment (Prod/UAT/DEV) • Application Name • Geographic Information (East vs West datacenter, office location, etc.) • AWS Region • Business Unit Highest level components should group the data how it is most often search together: Prod/Web/Apache/Access Dev/Web/Apache/Access Prod/DB/MySQL/Error Dev/DB/MySQL/Error Web/Apache/Access/Prod Web/Apache/Access/Dev DB/MySQL/Error/Prod DB/MySQL/Error/Dev
  • 11.
    Sumo Logic confidential WhatData can I Analyze? Option 1 Explore your Collectors Option 2 Search for source categories
  • 12.
    Sumo Logic confidential Operators Charts 2 Search& Analyze Search & Analyze Activity 2 1. To open a query, at the top, click +New and select Log Search 2. At the top query builder paste the following code (I will put in chat for you to copy) _sourceCategory=Labs/Apache/Access and "Mozilla" | parse ""GET *" * " as url,status_code | where status_code matches "5*" | count by status_code | sort by _count 1. You can modify the time to Last 60 minutes and click Start 2. You will have a table displayed and you can click on other chart types
  • 13.
    Sumo Logic confidential DataAnalytics ⇨ Query Syntax Syntax: metadata Keywords and operators, separated by pipes, that build on top of each other parse filter aggregate format keywords _sourceCategory=Labs/Apache/Access and "Mozilla" | parse ""GET *" * " as url,status_code | where status_code matches “5*” | count by status_code | sort by _count | limit 3
  • 14.
    Sumo Logic confidential DataAnalytics ⇨ Query Syntax Syntax: Use metadata and keywords to narrow your search scope Results keyword metadata keyword | parse | filter | aggregate | format metadata + keywords _sourceCategory=Labs/Apache/Access and "Mozilla"
  • 15.
    Sumo Logic confidential DataAnalytics ⇨ Query Syntax Syntax: Extract meaningful fields to provide structure to your data Parse Anchor: | parse " *@* " as user,domain Parse Regex: | parse regex "(?<src_ip>d{1,3} .d{1,3}.d{1,3}.d{1,3})” Other Parse Operators: csv, json, keyvalue, split, xml Learn more: Parse Operators | parse | filter | aggregate | format metadata + keywords
  • 16.
    Sumo Logic confidential DataAnalytics ⇨ Query Syntax Syntax: Further filter results using your extracted fields where operator: | where !(status_code=304) in operator: | if(status_code in("501","502"), "Error","OK") as code_type Other Filter Operators: join, lookup, matches, filter, isEmpty, isNull, isBlank Learn more: Filter operator example | parse | filter | aggregate | format metadata + keywords
  • 17.
    Sumo Logic confidential DataAnalytics ⇨ Query Syntax Syntax: Evaluate messages and place them into groups count operator: | count by status_code avg operator: | avg(size) by src_ip pct operator: | pct(filesize,75) by _sourceHost Other Aggregation Operators: sum, count_distinct, stddev, min, max Learn more: Aggregation operators | parse | filter | aggregate | format metadata + keywords
  • 18.
    Sumo Logic confidential DataAnalytics ⇨ Query Syntax Syntax: Format to display desired results succinctly top operator: | top 5 src_ip by avg_size fields operator: | fields src_ip, avg_size transpose operator: | transpose row src_ip column url Other formatting Operators: format, formatdate, limit, sort Learn more: Trends over time using transpose | parse | filter | aggregate | format metadata + keywords
  • 19.
    Sumo Logic confidential AdvancedAnalytics Geo Lookup _sourceCategory=Labs/Apache/Access | lookup latitude, longitude from geo://default on ip=src_ip | count by latitude, longitude Outlier _sourceCategory=Labs/Apache/Access and status_code=404 | timeslice 1m | count(status_code) as error_count by _timeslice | outlier error_count Predict _sourceCategory=Labs/Apache/Access | timeslice 5m | count as requests by _timeslice | predict requests by 5m forecast=12 Log operators Cheat Sheet: https://help.sumologic.com/05Search/Search-Cheat- Sheets/Log-Operators-Cheat-Sheet
  • 20.
    Sumo Logic confidential AdvancedAnalytics Find the “needle in the haystack” by identifying patterns. Compare today’s patterns with patterns in the past. _sourceCategory=Labs/snort | logreduce _sourceCategory=Labs/snort | logcompare timeshift -24h LogReduce LogCompare
  • 21.
    Sumo Logic confidential Alerts Dashboards 3 Visualize& Monitor Visualize & Monitor Activity 3 1. To create a dashboard, in the middle right click Add to Dashboard 2. In the popup window under Panel Title enter Track 500s under Dashboard enter Apache_db_<your initials###> 1. At the bottom of the popup click Add 1. To create an alert, in the left navigation pane click Manage Data> Alert 2. In the upper right corner click Add and select New Monitor
  • 22.
    Sumo Logic confidential Monitoring- Dashboards • Each Panel processes results from a single search • Drill down into corresponding query or link to another Dashboard • Live Mode: provides live stream of data • Use Dashboards as templates with Filters
  • 23.
    Sumo Logic confidential SumoLogic Data Flow Alerts Dashboards 3 Visualize & Monitor Operators Charts 2 Search & Analyze Collectors Sources 1 Data Collection What else may we address for you?
  • 24.
    Sumo Logic confidential •Onboarding Checklist https://help.sumologic.com/01Sta rt-Here/Onboarding_Checklist • Take the training https://www.sumologic.com/learn /training/ • Read the docs https://help/sumologic.com Want to learn more?
  • 25.
  • 26.
    Sumo Logic Confidential SumoLogic Confidential Which Topic would you like covered next? (Single Choice) Answer 1: Logreduce Operator Answer 2: New Dashboards Answer 3: Best Practices for Search Queries Answer 4: Partitions Answer 5: Collector Setup Optimization
  • 27.
    Sumo Logic confidential LogReduce– Foundation for Anomaly Detection Reduce hundreds of thousands of pages of results into a single page of meaningful patterns. Under the Covers: • LogReduce deconstructs log messages into their most basic patterns to facilitate overall behavioral analysis – at the printf level • Log messages are converted into unique hashed signatures – these signatures are the building blocks of anomaly detection – Events that occur more than others (e.g. errors flooding your logs) – Events that occur very infrequently but are important (e.g. rare exception) • Edit each signature to tailor each experience • Benefit from machine learning that improves over time based on your data and activity
  • 28.
    Sumo Logic confidential NewDashboards • New charts, like Honeycomb • Full control over look and feel with JSON • Build panels directly in the dashboard • Advanced filtering and metrics query building • Basic charts, like time series and categorical • Few color and font choices • Panels created from search and metrics tabs • Limited filters and queries • Still supported Classic Dashboard New Dashboard About New dashboards: https://help.sumologic.com/Visualizations-and- Alerts/Dashboard_(New)/About_Dashboard_(New)
  • 29.
    Sumo Logic confidential BestPractices for Search Queries
  • 30.
  • 31.
  • 32.