This document discusses configuring Elasticsearch to centralize logging from multiple components. It describes how Elasticsearch can be used to collect, search and export logs in near real-time from any location. It then provides instructions on updating application configuration files to define the Elasticsearch endpoint and send logs to Elasticsearch. Admins can then view centralized logs through Kibana.

1. It Shore Beats Working:
Configuring Elasticsearch to
get the Most out of Cloud
Logging

2. Private and Confidential – Copyright 2019
What’s in it for You?
ElasticSearch
Centralized
logging from a
remote
endpoint.
Easily export
relevant log
data.
Sequential
logging from
all
components.
Fast, near
real-time
searching
Easily
accessible
from any
location.

3. Presentation TitlePrivate and Confidential – Copyright 2019
Private and Confidential – Copyright 2019
Private and Confidential – Copyright 2019
What is Elasticsearch?
Open source distributed, RESTful search and analytics
engine capable of solving a growing number of use
cases.
• Real-time Full-text Searching capability
• Supports Aggregations
• Distributed and Scalable functionality
• Highly Available and Fault Tolerant
• Inverted index structure provides fast document
retrieval

4. Presentation TitlePrivate and Confidential – Copyright 2019
Private and Confidential – Copyright 2019
Private and Confidential – Copyright 2019
Centralize Logging with Elasticsearch
https://www.elastic.co/downloads/
Log
Query

5. Private and Confidential – Copyright 2019
Elasticsearch Dedicated System Requirements
• 64 GB RAM – Recommended
• (16-64 GB) 50% Java heap
Memory
• 2-8 Cores
• Concurrency outperforms clock speed
CPU
• SSD
• Use fastest affordable disk configuration possible – Do not use NAS
DISK
• Gigabit Ethernet (1GbE)
Network

6. Integrating
Ipro with
Elastic

7. Private and Confidential – Copyright 2019
Centralize Logging with Elasticsearch
Find Installed / Registered components
SELECT DISTINCT
[Name]
,[HostName]
,[IpAddress1]
FROM [ADDConfig].[Enterprise].[InstalledComponent]
Grid formatted log information includes:
• date/time
• machine name
• environment
• component
• type
• event status
When the log row is clicked, detailed error information appears

8. Private and Confidential – Copyright 2019
Configuring Elasticsearch
Current out of box
configuration is NLOG
Each application
component requires a
config file update
• C:Program Files (x86)IPRO Tech
• C:Program FilesIpro Tech
The configuration files are
located in the component’s
installation directory
• C:ProgramDataIPRO
Techconfiguration
Some components require
an additional log
configuration update in:
ADD System needs to be
supplied the Elasticsearch
endpoint

9. Private and Confidential – Copyright 2019
Updating the App.exe.config Files processing
Inside the Config
Find the
 <connectionStrings>
 element
Locate the attribute
<add
name=”ElasticSearch”
Update the
connectionString=
property to
(http://myelastichost:9200)

10. Private and Confidential – Copyright 2019
Updating the App.exe.config Files processing
Inside the Config
Find the
 <factoryAdapter>
 element
Locate the attribute
<add key=”target”
value =“elastic” />
Uncomment the line
to enable logging to
elastic

11. Private and Confidential – Copyright 2019
Updating the App.nlog.config Files processing
Inside the config,
locate the <rules>
element
Add a new attribute
<logger name=“*”
Then add the
following properties
minlevel=“Trace”
writeTo=“elastic” />

12. Private and Confidential – Copyright 2019
Updating the App.exe.config Files review
Inside the Config
Find the
 <rules>
 element
Locate the attribute
<logger name
Set
writeTo=“elastic”

13. Private and Confidential – Copyright 2019
Updating the App.exe.config Files review
Inside the Config
Find the
 <ConnectionStrings>
 element
Locate the attribute
<add
name=“ElasticSearch”
Update the
connectionString=
property to
(http://myelastichost:9200)

14. Private and Confidential – Copyright 2019
Choose System
from the main
menu.
In the left
pane of the
System page,
click Settings.
Under
Configurations
in the
Elasticsearch
Endpoint field,
enter the
Elasticsearch
URL.
Click the
corresponding
Save button.
Define Elasticsearch Endpoint

15. Private and Confidential – Copyright 2019
View Log Information
Choose
System
Select Logs

16. Private and Confidential – Copyright 2019
View Log Information Cont.

17. Let’s Have a
Looksee

18. Presentation TitlePrivate and Confidential – Copyright 2019
Private and Confidential – Copyright 2019
Private and Confidential – Copyright 2019
You know, for search.
• Leveraging Elasticsearch to CENTRALIZE
LOGGING for all components
• Elasticsearch is the heart of the Elastic Stack
and is the ONLY REQUIRED COMPONENT
• The components of the Elastic Stack include:
Elasticsearch
Logstash: Document Enrichment
Kibana: Visualization and Search UI

19. Private and Confidential – Copyright 2019
Logstash
Open source, server-side data processing pipeline that ingests data
from a multitude of sources simultaneously, transforms it, and then
sends it to your favorite “stash.” Elastic Search!
•Derive
structure from
unstructured
data with grok
•Decipher
geographic
coordinates
from IP
addresses
Ease overall
processing,
independent of
the data
source, format,
or schema.

20. Presentation TitlePrivate and Confidential – Copyright 2019
Private and Confidential – Copyright 2019
Private and Confidential – Copyright 2019
The Elastic Stack
Ipro
Log
Files
Ipro
Log
Files
Ipro
Log
Files
Collect, parse,
and transform
incoming data
Ports: 5044
Logstash output
is indexed for
searching
Ports: 9200
Logstash Elastic Search
Visualize and Query
Your Data with
Kibana (Ports: 5601)
View and Search
Ipro System Logs

21. Private and Confidential – Copyright 2019
Streaming Discovery
Log Snippet:
This is the target line in the log. A Grok filter must be added to the logstash
.conf file in order to field this data.

22. Private and Confidential – Copyright 2019
Configure Logstash with Grok Filters
input {stdin{}
file{path => "//Machine15/C$/ProgramData/IPRO Tech/logs/Ipro.eCapture.CurrentDiscovery.Service[Premium EDD Driver 1].log"}}
filter { grok {match => {"message" => " AProcessing Rate = %{NUMBER:ProcessingRate:float} GB/hour "} }}
output { elasticsearch { hosts => ["Machine001:9200"] }
stdout {}}
input {
stdin{}
}
output {
stdout {}
}
Default Configuration: Logstash.conf
Custom Configuration: Logstash.conf
https://www.elastic.co/guide/en/logstash/current/plu
gins-filters-grok.html#plugins-filters-grok-match
http://grokconstructor.appspot.com/do/construction
Grok Pattern Resources:

23. Private and Confidential – Copyright 2019
Kibana