SENTINL Presentation Slides (Amsterdam DevOps)

Alarm and Reporting App for Kibana & Kibi
Presented by Lorenzo Mangani
CEO QXIP BV, Amsterdam NL
http://qxip.net
Version 4.6 - Revision A
Introducing
SIREN SENTINL

WATCHING YOUR DATA 24/7/365
What is SENTINL?
SENTINL is an App Plugin extending Kibana or KiBi with Alerting and Reporting functionality
… 100% made in NL as the name suggests!
“Designed to monitor, validate and inform users and systems on data series changes using
standard or join queries, programmable result validators, transformers and templates to
send out notifications using a variety of configurable actions reaching users, interfacing with
remote APIs for data and commands, generating new Elasticsearch documents, arbitrary
metrics towards any other platform, planting triggers for itself to use and so much more. “
COUGH **FREE WATCHER ALTERNATIVE**
“I always feel like,
somebody’s
watching me…”
Rockwell, 1984

QXIP BV is an R&D specialized in VoIP and RTC Packet Capture, Monitoring and Analytics, but
experienced in many realms. Elasticsearch has been one of our customer’s favourite picks
and we created a pair of plugins to make their operational life easier and cheaper:
● ELASTICFENCE: Free Auth for ES + Kibana 2.x/5.x
● KAAE/SENTINL Free Alerts for Kibana 2.x/5.x
QXIP and SIREN Solutions decided to join forces to enhance the usefulness and scope of
SENTINL, which is not an overnight clone and leverages the DNA of our past experience:
SENTINL is a rebranded Kibi driven evolution of KAAE
KAAE was a Kibana Alerting App, late reincarnation of ELASTICWATCH
ELASTICWATCH was an early “watcher-like” tool for Elasticsearch featuring modular pipelines
validators, processors and notification capabilities for QBANA
QBANA was an ancient Kibana 3 fork, specialized with support for network monitoring, deep
packet inspection and featured with additional visualizers
… and the rabbit hole gets deeper!
BITS OF HISTORY
Possibly due to its
name, early KAAE only
ever received
contributions from
China and Japan

Elasticsearch & Kibana provide many unique features and enables integrators to realize
unique Data Intelligence creatures. With such power, being able to automate workflows
and get notified with data results and detections quickly becomes a critical requirement.
Elastic provides this feature with Watcher, an ES plugin part of the commercial X-Pack
ES Watcher lives in Elasticsearch and requires installation + license on all cluster nodes.
SENTINL provides similar functionality as an App executing within Kibana 4.x/5,x,
leveraging all of its native features such as secure client for queries and extends UI for
managing configuration, scheduling and handling executions of user Alerts and Reports.
SENTINL is also transparent to the Elasticsearch cluster(s) it monitors, appearing as a
regular client and requiring no complex installation, restarts and no licensing fees.
Powered by the many I/O modules the Node.JS community offers, SENTINL usage is not
limited to Elasticsearch and its capabilities can easily be extended to fully interface with
third party data sources and platforms for ingress and egress data.
So, How does it work? The main actors in SENTINL are called WATCHERS >>>
ELK + SENTINL: WHY?
SENTINL retains
compatibility with
KIBANA with the
exception of JOIN
query features

“ . . . WHAT IS A WATCHER ?”

Hello, Watchers!
SENTINL allows automation of recurring “questions” (as queries) by using Watchers
HIT COUNT PER HOUR
QUESTION: How many hits does index X receive hourly?
WATCHER: query index and return count of hits in last hour
ACTION: Notify with number of Hits per hour
METRIC THRESHOLDS
QUESTION: Is any of my monitored metrics surpassing a certain value?
WATCHER: query index and type for specific values, aggregated by an arbitrary field.
ACTION: Notify with aggs bucket details every time a threshold is surpassed or spike anomaly detected.
BLACKLISTS HITS
QUESTION: Is any of my users trying to reach blacklisted destinations?
WATCHER: query firewall logs comparing destination IPs to a blacklist.
ACTION: Notify admin via email if any IP >= 10 matches returned
FAILED LOGINS
QUESTION: Are there recurring failure attempts authenticating users on my network?
WATCHER: query active directory logs for login failures in last hour and compare to user index. .
ACTION: Notify admin via webhook if >= 10 matches returned
LEAK DETECTION (chain)
QUESTION: Are there any public leaks about my data I was not aware of?
WATCHER: query for user emails included in published leaks ingested from third parties.
ACTION: Save hits in secondary result Index. Notify via email if leak was not known in a secondary Watcher

1. Watcher Anatomy
A Sentinl Watcher (or just Sentinel) structure:
➔ Trigger Schedule
When and How to run the Watcher
➔ Input Query
What Query or Join Query to Execute
➔ Condition Script
How to conditionally Analyze Response
➔ Transform Script
How to Adapt or Post-Process data
➔ Actions
How to Notify users about this event

1. Watcher Anatomy
➔ Input Query
➔ Actions
Example:
10:15am every day
'at 10:15 am'
Every 5 minutes every day
'every 5 mins'
Every 5 minutes every day
'every 5 mins'
Every Sunday
'on the first day of the week'
On the last day of every month
'on the last day of the month'
Every 5 minutes on Weekends
'every 5 mins every weekend'
TRIGGER SCHEDULE
The schedule defines a set of constraints that must be met to
execute a saved watcher. Any number of constraints can be
added to a single schedule, and multiple rules can be
combined to achieve complex intervals, programmed using
simple text expressions using the NodeJS later module.

1. Watcher Anatomy
➔ Input Query
➔ Actions
INPUT QUERY
The input parameter is the key element of a watcher, and
defines a dynamic range index query feeding the circuit.
The input field accepts any standard Elasticsearch query
including server side scripts in supported languages and
fully supports the Siren Join capabilities out of the box.
"input" : {
"search" : {
"request" : {
"indices" : [ "<mos-{now/d}>" ],
"body" : {
"query" : {
"filtered" : {
"query": {
"query_string": {
"query": "mos:*"
}
},
"filter" : {
"range" : {
"@timestamp" : {
"from" : "now-5m"
}
}
}
}
}
}
}
}
}

1. Watcher Anatomy
➔ Input Query
➔ Actions
CONDITION SCRIPT
The condition script is the “entry gate” into the processing
pipeline of a Watcher and determines its triggered status.
The script simply expects a boolean condition, and can be
as simple as a hit counter or as complex as an aggs climber.
On true condition, the pipeline will proceed further.
On false condition, the pipeline will stop until its next turn.
BONUS: Scripts can be saved and used across Watchers!
"condition": {
"script": {
"script": "payload.hits.total > 100"
}
}

1. Watcher Anatomy
➔ Input Query
➔ Actions
TRANSFORM SCRIPT
The transform script is the wild member of the family and
can be used to inject simple or complex logic into the
pipeline before delivery to actions using pure javascript.
From converting format types, through generating brand new
payload keys and interpolating data, transform is the way up.
The script expects a boolean condition to trigger actions.
A false condition can be forced to stop the execution.
BONUS: Transforms can be saved and used across Watchers!
"transform": {
"script": {
"script": "payload.newvar = payload.aggs.some.values['95.0']"
}
}

1. Watcher Anatomy
➔ Input Query
➔ Actions
ACTIONS
Actions are used to deliver any results obtained by a
Watcher to users, APIs or new documents in the cluster.
Multiple Actions and Groups can be defined for each.
Actions use the {{ mustache }} logic-less template syntax,
and work by iterating arrays and expanding tags in a
template using values provided in the response payload.
Actions require configuration via the Kibi config yaml in order
to be enabled and to provide each module with its settings.
"actions" : {
"email_admin" : {
"throttle_period" : "15m",
"email" : {
"to" : "mos@qxip.net",
"from" : "sentinl@qxip.net",
"subject" : "Low MOS Detected: {{payload.aggregations.avg.value}}",
"priority" : "high",
"body" : "Low MOS Detected:n {{payload.aggregations.avg.value}}"
}
}
}

Zoom on Actions!
SENTINL features several built-in actions used to deliver watcher results to endpoints.
● EMAIL ( Plain Text )
● EMAIL HTML ( PDF, Image Reports )
● GENERIC WEBHOOK ( Any API or Webservice )
● SLACK WEBHOOK ( Using Auth Hook )
● PUSHAPPS ( IOS/Android Notifications )
● ELASTICSEARCH ( via WebHook )
● NAGIOS NRDP ( via WebHook )
Actions can easily be extended with new notification methods by adding node modules.
Webhooks can be used for just about any integration using JSON, XML or custom formats
including BULK inserts into Elasticsearch and other web based service.
Output format flexibility is achieved by supporting {{ mustache }} logic-less templates.
{{ }}

Mustache: Logic-Less Templates
Mustache allows users to arrange and compose notification bodies using logic less
templates inline, providing easy access to use and display complex object dynamically.
Here’s a typical Mustache example using values and iterating arrays:
Mustache template:
Hello {{name}}
Value is {{value}}
{{#list}}
* I see a {{item}}
{{/list}}
GoodBye!
Source JSON Payload:
{
“name” : “Someone”,
“value” : 100,
“list” : [
{ “item”, “Pen” },
{ “item”, “Pencil” }
]
}
Resulting Body:
Hello Someone
Value is 100
* I see a Pen
* I see a Pencil
Goodbye!
{{ }}
+ =

“WHAT IS THE LIFECYCLE OF WATCHERS ?”

SENTINL WATCHER LIFECYCLE
Execute Action
Example:
Take a screenshot of the
dashboard and send email
Execute Request
Example:
Check how many error
documents match a query
every 5 minutes
Validate Response
Example:
If more than 10 events exist in
the last 5 minutes, proceed
At the set mark each WATCHES will execute its query, validate results and if true, execute actions

Execute Action
New JSON Document
Execute Request
Index #1
Validate Response
Transform to JSON _Bulk
INSERT NEW DOCUMENT TO INDEX #2
Execute Action
New JSON Document
Execute Request
Index #2
Validate Response
Transform to JSON _Bulk
SENTINL WATCHER SIDECHAINS
CREATE NEW DOCUMENT IN INDEX #3
Multiple WATCHES can be chained, analyzing each other’s output to answer complex questions

Execute Action
DELETE Watcher
Execute Request
Index #1
Validate Response
SENTINL WATCHER SELFCARE
DELETE WATCHER
Watchers can even self-interact Actions via Elasticsearch API deprovisioning post execution.
CREATE WATCHER
NOTIFY ADMIN

“HOW TO TURN IDEAS INTO WATCHERS ?”

WATCH MAKING BUSINESS
Creating Watchers is not always an easy task and requires decent knowledge of the Elasticsearch query
language as well as a clear understanding of the problem one’s trying to solve in the first place.
Just like real watches, an intricate and beautiful mechanism can take patience and experience to run.
Watchers can be created and modified using the SENTINL user interface expert mode editor, or fully
managed directly via the SENTINL API as JSON objects from KiBi integrated applications and tools.
The SENTINL App can be launched via the Kibi App Switcher or by using the SPY plugin helper.

TURN USERS INTO ALERT SCIENTISTS
SENTINL is designed to provide both Expert and Standard editing capabilities.
Some Users cannot write full native Elasticsearch Queries just yet? Not a Problem!
The included SPY Plugin can turn any Visualization into a Watcher skeleton:
Tip
The project Wiki
contains numerous
working examples
to get started with!
Visit it often to
learn more about
new hot recipes!

When creating new watchers via SENTINL UI a Wizard is provided to guide users through the steps:

Create or Copy-Paste the Watcher Query used to produced results, with dynamic time-range:
Input queries can be created, saved and reused across different SENTINL Watchers

Define the Condition Script acting as a gate to further process results:
Condition Scripts can be created, saved and reused across different SENTINL Watchers

Define the Transform Script used to manipulate or format data before sending to an action set:
Transform Scripts can be created, saved and reused across different SENTINL Watchers

Define the Actions for delivering results and notifications to one or multiple gateways/recipients:

TRUSTED SCHEDULING, MANAGEMENT AND SECURITY
SENTINL core actions are safely executed by the KiBi server via its internal Elasticsearch client.
Authentication is automatically inherited from the core and watchers are saved in the Cluster.
Watchers running within the Enterprise version of KiBi also feature index and document level access
control to the cluster data using the Security and Access Control features provided by Search Guard.
Upcoming releases will include per user and group Watcher indices and their partitioned management,
as well as GunDB based event distribution to handle watchers jobs across multiple running instances.

ALERT & REPORT EXECUTION HISTORY
SENTINL keeps track of executed Watches and logs all Alerts and Reports generated by the system in
Elasticsearch indices, available for further analysis and interaction via regular widgets and dashboards.
The full Alarm and Report History is accessible and searchable directly via the User Interface or via
custom visualizers. Preferences for indices and type naming are available via kibi configuration yaml.
Tip
Sentinl fully leverages the
Elastic Query Language and
EE Joins.
Existing knowledge can be
easily deployed to create
complex chains of Alarms
and Events.

“LET’S SEE SOME ACTION USE CASES”

1. Example: Email + Slack Notification
"actions": {
"email_admin": {
"throttle_period": "15m",
"email": {
"to": "alarm@mycompany.com",
"from": "sentinl@mycompany.com",
"subject": "SENTINL Alarm",
"priority": "high",
"body": "Found {{payload.hits.total}} Events"
}
},
"slack_admin": {
"slack": {
"channel": "#kibi",
"message": "SENTINL Alert! Found {{payload.hits.total}} Events"
}
}
}

2. Example: Elasticsearch (Bulk Insert)
"actions": {
"ES_bulk_request": {
"webhook": {
"method": "POST",
"host": "elasticsearch.mycompany.com",
"port": 9200,
"path": ":/_bulk",
"body":
"{{#payload.aggregations.metrics.buckets}}
{“index”:{“_index”:”agg_requests”,”_type":"data”}}n
{“url”:”{{key}}”,”count”:”{{doc_count}}"}n
{{/payload.aggregations.metrics.buckets}}nn",
"headers": {
"Content-Type": "text/plain; charset=ISO-8859-1"
}
}
}
}

3. Example: Visual Report (Image or PDF)
"report": {
"to": "reports@mycompany.com",
"from": "alarm@mycompany.com",
"subject": "SENTINL Dashboard Report",
"priority": "high",
"body": "Sample SENTINL Screenshot Report",
"snapshot": {
"res": "1280x900",
"url": "http://www.google.com",
"path": "/tmp/",
"params": {
"delay": 5000
}
}
}

4. Example: Nagios NRDP (XML)
"my_webhook" : {
"throttle_period" : "5m",
"webhook" : {
"method" : "POST",
"host" : "nagios.mycompany.com",
"port" : 80,
"path": ":/nrdp",
"body" : "token=TOKEN&cmd=submitcheck&XMLDATA=
<?xml version='1.0'?><checkresults>{{#ctx.payload.aggregations.mos.buckets}}
<checkresult type='host' checktype='1'>
{{#by_group.buckets}}
<hostname>{{key}}</hostname>
<servicename>MOS</servicename><state>0</state>
<output>MOS is {{avg.value}}</output>
{{/by_group.buckets}}</checkresult>
{{/ctx.payload.aggregations.mos.buckets}}
</checkresults></xml>"
}
}

4. Example: Failed Login Attempts
"condition": {
"script": {
"script": "payload.newlist=[];var match=false;var threshold=10;
var start_level=2; var finish_level=3;
var first=payload.aggregations[start_level.toString()];
function loop_on_buckets(element,start,finish,upper_key){element.filter(
function(obj){return obj.key;}).forEach( function ( bucket ) { if (start == finish - 1) {
if (bucket.doc_count >= threshold) { match=true; payload.newlist.push({
line: upper_key + bucket.key + ' ' + bucket.doc_count}); } } else {
loop_on_buckets(bucket[start + 1].buckets, start + 1, finish, upper_key + ' ' +
bucket.key); } }); } var upper_key = '';
loop_on_buckets(first.buckets, start_level, finish_level, upper_key);match;"
}
},
"actions": {
"email_admin": {
"email": {
"to": "sentinl@localhost",
"from": "alerts@localhost",
"subject": "Sentinl - user failing logins",
"body": "User in AD group 'XXX’ Users failed >=10 logins in 1hr
Events:n{{#payload.newlist}}nn{{line}}{{/payload.newlist}}" }
}
}

5. Example: ATLAS Anomaly Detection
The Project wiki provides many examples including an ATLAS statistical anomaly detector
Use-Case Scenario:
● varnish-cache server as Frontend-LB and caching Proxy
● backends are selected based on their first_url_part
● backends dynamically added or removed by development teams (or applications)
DevOps Problem:
Monitoring 95th percentile values of consolidated backend runtimes conceals issues located with
a specific backend server, while drawing stats for each server would results in too much data.
Solution:
To implement the ATLAS algorithm in SENTINL using two watchers:
● WATCHER 1: Collect a most surprising req_runtime of every backend for every hour to ATLAS
● WATCHER 2: Iterates every 5 minute over the ATLAS index to find anomalies to report if the
average runtime of the last bucket is grater than upper bound of the std_dev.

SENTINL Project Roadmap
Here are the Highlights from our current SENTINL RoadMap:
Complete migration to ELK 5.x (beta is out!)
Distribution of Events and Tasks in cluster via GunDB
○ Real-time trigger status and updates with no load on ES Cluster
○ Sequential Actions, Dynamic Watcher generation/removal
Support for External Queries in WATCHERS
○ Cassandra
○ InfluxDB
○ MySQL/PgSQL
○ Generic REST API

THAT’S ALL, FOLKS!
SOUNDS INTERESTING?
Join the hundreds already using it and help us with suggestions, requests and bug reports!
SENTINL is super easy to install in Kib% and requires no changes in Elasticsearch:
https://github.com/sirensolutions/sentinl
Need help getting started?
https://github.com/sirensolutions/sentinl/wiki
THANKS FOR YOUR TIME
Got Questions? Ask Away!

SENTINL Presentation Slides (Amsterdam DevOps)

More Related Content

Recently uploaded

Featured

SENTINL Presentation Slides (Amsterdam DevOps)