InfiniFlux Collector is the processthat collects various forms of log files from remote servers by using log collector and insert intoInfiniFlux DB.

1. InfiniFlux Collector
www.infiniflux.com

2. Table of Contents
What is Collector
Create and Start Collector
Setup Client - log aggregator
Setup Client - rsyslog
Setup Client - logstash
1
2
3
4
5
Setup Client with shell script6
Tag7
Add custom log8

3. What is Collector
3
InfiniFlux Collector is the process that collects various forms of log files from remote servers by
using log collector and insert into InfiniFlux DB.
• Syslog
• Apache access
• Tomcat server
• Java, PHP, etc
• Application log
• InfiniFlux log
• Custom log
</>
</>
</>
Send via TCP InfiniFluxAppend
Client
iflux
collector

4. Create and Start Collector
4
InfiniFlux sever must be operated before creating and executing collector.
• Create and execute InfiniFlux DB
$ ifluxadmin -c  Create InfiniFlux DB
$ ifluxadmin -u  Execute InfiniFlux server
• Create and execute collector
$ ifluxcollector –c –p 5657  Create collector through 5657 port
$ ifluxcollector –u –p 5657  Execute collector through 5657 port
Default port number: 5657
Stand by in order to receive TCP data through designated port number.

5. Setup Client - log aggregator
5
Currently InfiniFlux provides two types of log collectors; rsyslog and logstash.
- Rsyslog reads and sends log files, and immediately process when events occur.
- Logstash reads and sends log files.
iflux
collectorSend via TCP
Log Aggregator
File rsyslog
Read
File logstash
Read
Event rsyslog
Raise

6. Setup Client - log aggregator
6
Below are the values to be set in the client file that sends logs.
There are four required properties and two selected properties.
Collector host
(HOST)
Collector port
(PORT)
Table name
(TABLE)
Tag name
(TAG)
Path of log file
(FILE_PATH)
Port that receives data
(RECV_PORT)
• Required Property
• Selected Property

7. Setup Client - logstash
7
This is how to send logs to collector by using logstash.
• Verify versions of Java
It is required to verify the version of Java since logstash can be operated on version of Java 7 or more.
$ java -version
• Checkout and download logstash
• Check whether logstash is installed, and download it from the website below if not installed.
https://www.elastic.co/products/logstash
• Create a configuration file of logstash.
Name a configuration file of “apache_access” as “apache_access.conf”.
Detailed information on configuration file is described on the next page.
• Check if the configuration file is valid.
$ logstash –f apache_access.conf –configtest
• Execute logstash.
$ logstash –f apache_access.conf

8. Setup Client - logstash
8
This is the template and example of logstash.
Set values of FILE_PATH, TABLE_NAME, TAG, HOST, and PORT to your environment.
input {
file {
path => “FILE_PATH”
}
}
filter {
mutate {
add_field => {
“table” => “TABLE_NAME”
“tag” => “TAG”
}
}
}
output {
tcp {
codec => line {
format => “[[table:%{[table]} tag:%{[tag]}
message:%{[message]}]]”
}
host => “HOST”
port => “PORT”
}
}
input {
file {
path => “/var/log/httpd/access_log”
}
}
filter {
mutate {
add_field => {
“table” => “apache_access_table”
“tag” => “apache_access”
}
}
}
output {
tcp {
codec => line {
format => “[[table:%{[table]} tag:%{[tag]}
message:%{[message]}]]”
}
host => “192.168.0.10”
port => “5657”
}
}
Template Example

9. Setup Client - rsyslog
9
This is how to set configuration file when log file is sent by using rsyslog.
• Rsyslog is basic syslog protocol so that, in general, most of Linux has it, but if not, it can be downloaded
from the website below.
http://www.rsyslog.com/
• Create a configuration file for the log.
Create the configuration file to “/etc/rsyslog.d” directory in the form of <name of configuration file>.conf.
• Re-execute rsyslog.
Create a configuration file and re-execute rsyslog in order to reflect the newly created configuration file.
$ sudo service rsyslog restart
• Verify data entry.
To verify whether the data is properly inserted, select data in the table by using ifluxsql.

10. Setup Client - rsyslog
10
This is the configuration file example of rsyslog.
Create the file in “/etc/rsyslog.d/syslog.conf.”
$template TABLE_NAME_Format, “[[table:TABLE_NAME tag:TAG message:%msg%]]n”
$ModLoad imfile
$InputFileName FILE_PATH
$InputFileTag TAG_file:
$InputFileStateFile stat-TAG
$InputFilePollInterval 1
$InputRunFileMonitor
If $programname == ‘TAG_file’ then @@HOST:PORT;TABLE_NAME_Format
if $programname == 'TAG_file' then ~
$template syslog_table_Format, “[[table:syslog_table tag:syslog message:%msg%]]n”
$ModLoad imfile
$InputFileName /var/log/syslog
$InputFileTag syslog_file:
$InputFileStateFile stat-syslog
$InputFilePollInterval 1
$InputRunFileMonitor
If $programname == ‘syslog_file’ then @@localhost:5657;syslog_table_Format
if $programname == 'syslog_file' then ~
Template
Example

11. 11
Shell scripts are provided for creating configuration files easily.
• Location of shell scripts
$ cd $IFLUX_HOME/collector/samples/conf
• There are three shell script files.
file-config-for-logstash.sh : script for setting up logstash
file-config-for-rsyslog.sh : script for setting up rsyslog
syslog-config-for-rsyslog.sh : script for setting up syslog file
• Create a configuration file by using scripts.
$ sudo ./file-config-for-rsyslog.sh <COLLECTOR_HOST> <COLLECTOR_PORT> <TABLE_NAME> <TAG>
<FILE_PATH>
$ sh ./file-config-for-rsyslog.sh 192.168.0.10 5657 apache_access_table apache_access
/var/log/httpd/apache_access
• Verify the created configuration file.
$ cd /etc/rsyslog.d
Created file: 192.168.0.10:5657_apache_access_table_from_file.conf
Setup Client with Shell Script

12. Tag
12
Tag type Description
syslog It is able to receive standard format of syslog which is created as a file.
apache_access It receives combined format of apache access log.
infiniflux
It receives trace log file of InfiniFlux.
Trace logs can be found in the “$IFLUX_HOME/trc” directory.
JSON
It dynamically process data in the form of JSON.
Data must be inserted in the form of {“key”:“value”}, and all the column will be created in the
VARCHAR format if a table is not created beforehand.
Unparse
When “unparsed” logs were received, it inserts the log in a VARCHAR column rather than
parsing it.
• Supported tag types
• To add a tag
• All the tags will be saved in the “$IFLUX_HOME/collector/regex” directory as files.
• The same name should be given to a file and tag, and need a declaration for a each column and regular expression should be
defined.
• To reflect modified information of tag file, re-execute the collector when contents of the file are changed.
Need to specified tags in order to separate log types, and parse log data that were inserted in
accordance with defined tags.
When there is a new kind of log entry, a new tag must be added as well.

13. Add Custom Log
13
Item Description
REGEX_NO
It determines nth token among the tokens that were specified in the regular expression.
To check the number of token, use ifluxregex utility.
NAME When there is no table to be inserted, a new one is created. “NAME” will name the new table.
TYPE It specifies data type of table column.
SIZE
It specifies the size of table column.
For the case of VARCHAR type, it needs to be specified precisely.
DATA_FORMAT If a column type set to “datetime”, it specifies the format of datetime of input log.
USE_INDEX
It specifies whether to create an index for the column. If it is 1, create an index.
For the formats of VARCHAR and text, it creates keyword index, and bitmap index for the rest.
Automatically, <table name>_idx_<field name> name will be created.
• COL_LIST
• REGEX
• Write regular expression for the log.
• Each log will be inserted as a sentence unit except a new line.
When there is a new kind of log is received, a new tag file for the log should be added in the
“$IFLUX_HOME/collector/regex” directory. A tag file is consisted of COL_LIST and REGEX.

14. 14
COL_LIST= (
(
REGEX_NO = 0
NAME = tm
TYPE = datetime
SIZE = 8
DATE_FORMAT="%b %d %H:%M:%S"
),
(
REGEX_NO = 4
NAME = host
TYPE = varchar
SIZE = 128
USE_INDEX = 1
),
(
REGEX_NO = 5
NAME = msg
TYPE = varchar
SIZE = 512
USE_INDEX = 1
)
)
REGEX="(([a-zA-Z]+)s+([0-9]+)s+([0-9:]*))s(S*)s+((?:[^0])*)"
Add Custom Log
This is the contents of syslog tag file.
Tag file can be found in the “$IFLUX_HOME/collector/regex/syslog”.

15. The World's Fastest
Time Series DBMS
for IoT and Big Data
www.infiniflux.com
info@infiniflux.com
InfiniFlux