Log insight technical overview customer facing (based on 3.x)

1. © 2014 VMware Inc. All rights reserved.
Log Insight Technical Overview
Log Insight 3.3
David Pasek
VMware TAM

2. vRealize Log Insight 3.0 - Agenda
• Overview
• Log Insight for vCenter
• Usability & Visualization Options
• Administration & Scale Details
• DEMO
CONFIDENTIAL 2

3. Log Insight Overview

4. Hybrid Cloud
(Private / Public)Physical Infrastructure
SOFTWARE-DEFINED DATA CENTER
Compute Network Storage
End-User Computing
Extensibility
Applications
Cloud Management Platform (CMP)
Virtualized Infrastructure
Introduction: Environment Landscape
VMware Logs
OS and
App Logs
200 ESXi Host + VMs = 200GB or 2B log events per day
Physical Infrastructure Logs

5. Primary Use Cases
Troubleshooting and Root Cause Analysis
• Follow the trail from vRealize Operations Manager to logs to get to root cause to an
observed problem
• Identify the needle in the haystack in real time when troubleshooting a problem
Monitoring
•Monitor metrics and events (performance & change) that are visible only in logs
•Identify problems proactively, ensure SLAs and comply to IT policies
Unstructured Data Warehouse
• Collect all the data in one place without the need for custom parsing, transformation of data
• Get full visibility across all your IT environment from a single place
5

6. vRealize Log Insight Overview
Intelligent Operations
•Enterprise Scale
•Predictive Analytics/Machine Learning for faster
problem resolution
Built for the Software Defined Data Center
•Base version now included with vCenter
•Insight into VMware products incl. NSX, vRealize
Automation, Horizon View
•Attractive pricing model for customers of all sizes –
not based on log volume
Unified Management
• Integration with vRealize Operations Management
Suite Inventory integration, alert notifications
Extensible
• Over 40 Content Packs Available
Operating
system
vSphere
System
statistics
Applications
Security
Infrastructure
Logs
Log Insight 2.0
Analyze
Discover
Search
Visualize
IT Operations
Security
Compliance
40B events 10 event types
…by machine learning
OverviewApp
App
CONFIDENTIAL

7. vRealize Log Insight for vCenter
vRealize Log Insight
Available with vCenter Server Standard
Intelligent Log Analytics for vCenter
• Free 25-OSI Log Insight pack per vCenter
Benefits of vRealize Log Insight for vCenter
• Powerful big data log management built for vSphere
• Includes all VMware Content Packs
• Extensive Log Management – Captures log data from
physical servers, network and storage devices, OSs,
applications, and more
• Intuitive on-the-fly keyword filtering and custom
dashboards
• Integration with vRealize Operations – Inventory
integration, 2-way Alert Visualization
NEW
The best real-time
management for SDDC
Security
A
p
p
Operating
System
vSphere
System
Statistics
Applications Other IT
A
p
p
Logs
Upgrade to Full vRealize Log Insight for:
• Extensibility – 3rd Party and Custom Content Packs such
as Microsoft, Cisco, EMC, NetApp (29+ available)
• Scalability – Cluster Support and Event Forwarding
• High availability, Archiving and SSL

8. Technical Overview

9. Log Insight Technical Overview
Cloud / Data Center
Log
Management
OS
Logs
VC
Logs
App
Logs
System
Stats
Security
Logs
API Syslog
Analyze
• Can analyze any unstructured time-series data,
configuration etc.
• Automatically identifies structures in the data
Scale
• Central, scale-out store (no-SQL) for all collected logs
• Configurable retention and archiving
• Maintenance free
Best for SDDC
• Queries, alerts, fields, charts
in the vSphere Content Pack

10. Intelligent Operations
Predictive Analytics
• Machine Learning based Automatic
Data Consolidation
• Intelligent data summarization
• Cluster similar messages together
• Automatic Schema extraction
• Automatically understand message
structure
• Intelligent automatic field extraction
Technical Overview

11. It’s like ‘Rosetta Stone’ for logs
Log Insight proactively learns:
from:
Then you can query it like a database!

12. Machine Learning
• Automatic event clustering
– Cluster similar events: reverse-engineer line of code that generated events
– Happens at ingestion time so zero impact on interactive analytics
– Example: Search retrieves 10,000, but summarized as just 10 event types
• Schema discovery
– Automatically understand the structure each event
– Automatic field extraction: “smart fields” defined for each event, including their data
types

13. vSphere Content Pack
• Ships out of the box
• Knowledge about ESXi and vSphere
logs as well as vCenter Alarms,
Events & Tasks
• It consists of: Queries, alerts,
dashboards, group templates, and
field extractions
• Divided into functional categories
– Including ESXi, Storage, and
vCenter Alarms
• vSphere and Content Pack
dashboards cannot be modified –
users can clone them into their
workspace

14. Visualize Log Data Using Dashboards
Run all queries in the list
Dashboard Filters
Launch into Interactive
Analytics
Types of Dashboards:
Personal
Shared by Admins
Content Pack
Choose Dashboards

15. Interactive Analytics
Fields with breakdown
charts
Query time range
Overview Chart:
By default: count of events
over time
Time bar length
Multiple
Aggregation
functions/analytics
Search Box and Query Builder
Events List

16. Interactive Analytics – Events List
• Content pack fields belong to a namespace (e.g. vSphere) so that they don’t
collide
• The system timestamp on the left is the arrival timestamp that Log Insight uses
• Clicking a field in the Events list or a bar in the overview chart list creates a
constraint
• The constraints can form a logical AND (match all) or logical OR (match any)
Message arrival timestamp,
can differ from the one
embedded in the message
Total matches
Standard syslog fields
(defined in syslog RFC)
CP and Integration fields

17. vRealize Log Insight – Usability & Visualization
Intelligent Visualizations & Extensibility
• Multiple-Function charts
• Chart Options
• Snapshots Visualization
• Share a Query
• Event Type Colorizing, Highlighting
Technical Overview
Smarter Visualizations
Snapshots
Visualization
Multi-Function
Charts

18. vRealize Log Insight – Multiple Function Charts
CONFIDENTIAL 20
Add additional
function to
chart

19. Overview Chart Visualizations
• Table View and Additional Visualizations
– Field Table
• Show/hide columns
• Add to Dashboard
– Additional chart types
• Column Chart
• Line Chart
• Area Chart
• Bar Chart
• Pie Chart
• Bubble Chart
• Data Table

20. vRealize Log Insight – Snapshots Visualization
CONFIDENTIAL 22
Create
Snapshot

21. vRealize Log Insight – Share a Query!
Log Insight 3.x
Auto-Shortened,
friendly URLs for
sharing

22. Event Types - Highlight and Colorize
• Easier visual analysis of events
• Purpose: Troubleshooting and RCA
• How it works: Select the gear icon to the left of any event and select Highlight/Colorize option
CONFIDENTIAL 25
Quickly Identify
Distinct Event
Types, Within
Events Tab

23. View Chart Data as a Table
26
Build data
visualizations
using a table
Choose Table to
change the
visualization
•

24. vRealize Log Insight – Administration & Scale
Highly Available, Simplified, and Scalable
• Ingestion and Query HA
• 12 Cluster Nodes, 48TB of live log data, 2.7 TB per day
• Integrated Load Balancer with Multiple VIPs
• Enhanced vSphere Integration
• APIs
• Authentication and Simple Query
• Agent & Agent Management
• Client-side event parsing
• Agent Groups for Centralized configuration
• Agent SSL Support
• Webhooks
• Rolling Upgrades & Rollback for Cluster Nodes
Technical Overview
Simplified and Powerful Administration
Agent
Configuration
Groups

25. vRealize Log Insight – Query HA
Log Insight 2.5 Log Insight 3.x

26. Integrated Load Balancer – Multiple VIP
30
Define
multiple VIPs
with ILB
Associate tags with
each VIP. Useful for
RBAC and Content
Packs
ILB Election
process is
unchanged

27. LI 3.x - Query HA
Log Search
Worker
Log Search
Worker
Log Search
Worker
Log Search
Worker
Log Search
Master
UI
Load Balancer
Client
VIP
UI Query Request
Log Search
Master
Log Search
Master
Log Search
Master
UI UI UI
Master
Log Search
Master
UI

28. LI 3.0 Query HA – Master Dies
Log Search
Worker
Log Search
Worker
Log Search
Worker
Log Search
Worker
Log Search
Master
UI
Load Balancer
Client
VIP
UI Query Request
Log Search
Master
Log Search
Master
Log Search
Master
UI UI UI
Master

29. Authentication API
• Use-case: Required to leverage the query API (discussed next)
35
Request Response
GET /api/v1/sessions HTTP/1.1
Host: localhost:9543
Accept: application/json
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
{
"userId": ”<user_guid>",
"ttl": <int>
}
POST /api/v1/sessions HTTP/1.1
Host: localhost:9543
Accept: application/json
Content-Type: application/json
{
”provider”: “<Local|ActiveDirectory>”,
"username": ”<username>",
"password": ”<password>"
}
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
{
"userId": ”<user_guid>",
"sessionId": ”<session_id>",
"ttl": <int>
}

30. vRealize Log Insight – Agent Groups
Predefined “Agent
Groups” configuration
can now be included in
content packs

31. vRealize Log Insight – Agent Groups (con’t)
CONFIDENTIAL 38
Define Agent
Configuration for
Group
Create New
or select
existing
Agent Group
Define Filter(s) to apply
configuration to specific
systems
Results
Based on
Defined
Filter(s)

32. vRealize Log Insight – Client-Side Event Parsing
CONFIDENTIAL 39
Example: CSV Log Parser
liagent.ini
Example: Configuration
[parser|myparser]
base_parser = csv
fields = field_name1,
field_name2, field_name3
delimiter = ";"
[filelog|some_csv_logs]
directory=D:Logs
include=*.txt;*.txt.*
parser=myparser
; define a parser section with an arbitrary name
; which can be referred from log sources
[parser|myparser]
; parser section refers to any defined parser such as csv, kvp, etc
base_parser = csv
; define extracted fields to include separated by commas
fields = field_name1, field_name2, , field_name4
; define delimiter to be used by the parser enclosed in quotes
Delimiter = “;”
; after defining parser, refer to it from winlog or filelog sources
[filelog|some_csv_logs]
; define source directory
Directory=csvsource1
; define log files to include
Include=*.txt
; refer to the parser name you defined above
parser=myparser

33. Integrations and
Extensibility

34. vSphere Integration
42
Add tags for vCenter
and ESXi events.
RBAC use case
Automatically add
unconfigured
Hosts
Choose ILB VIP

35. Better Together: vRealize Operations and vRealize Log
Insight
Leverage all your IT data for comprehensive visibility in one place
Structured Data
Metrics Alerts Events
VMware vRealize
Operations
Capacity, Performance and
Configuration Management Events
Launch in
Context
Unstructured Data
Logs Messages
VMware vRealize
Log Insight
Log analytics, aggregation,
and search
Public
Cloud

36. VMware vRealize Log Insight Extensibility
Highly Extensible
• Captures log data from physical servers, network and storage devices, OSs, applications, VMs, and hosts, and more
Log Insight Content Packs
• Encapsulate, pre-built dashboards and product-specific alerts from vRealize Log Insight
• Provide vendor specific guidance and insight into which logs really matter
Log Insight Marketplace
• Built into the UI or available at www.solutionexchange.vmware.com
Content Packs
Overview
• Operating System
• Application
• Network
• Storage
• SDDC
• Security
CONFIDENTIAL

37. Demonstration

38. Learn More
Try the Hands-on Lab.
Nothing to download!
47
Visit the website for
resources, 60-day free trial,
evaluation guide, and
purchasing information.
@VMLogInsight
www.vmware.com/products/vrealize-log-insight
vmware.com/go/vRealize-Ops-Insight-HOL
loginsight.vmware.com/
Website:
Hands-on Lab:
Log Insight Community:

39. Thank You