Kibana_Data_analyst_7.1.0.pdf

An Elastic Training Course
elastic.co/training
7.1.0
Kibana Data Analyst

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or
distributing without written permission is strictly prohibited
About This Training
• Environment
• Introductions
• Code of Conduct (https://www.elastic.co/community/codeofconduct)
• Agenda...
!2

Kibana Search
Kibana Fundamentals
1
Kibana Visualizations
Kibana Dashboards
Kibana Visual Builder
Kibana Management
Course Agenda
!3
2
3
4
5
6

Kibana Fundamentals
Module 1
Kibana Search
Kibana Dashboards
Kibana Fundamentals
Kibana Management

Topics
• Introduction to Kibana
• Discover Interface
• Aggregations
!5

Introduction to Kibana
Kibana Fundamentals
Lesson 1

The Elastic Stack
!7
INGEST VISUALIZE
INDEX | QUERY | AGGREGATE

Ingest: Logstash and Beats
• Logstash
‒ Server-side data processing
‒ Ingests data from multiple sources simultaneously (MongoDB,
PostgreSQL, Elasticsearch, ...)
‒ Parse, transform and prepare your data for ingestion
• Beats
‒ Single purpose data shippers
‒ Many flavors: Filebeat, Metricbeat, Packetbeat, Winlogbeat, ...
‒ Lightweight agents that send data from a machine to
Elasticsearch or Logstash
!8

Index: Query and Aggregations
• Elasticsearch
‒ Heart of the Elastic Stack
‒ distributed: easy to scale
‒ RESTful: easy to communicate with using APIs
‒ search, analyze and store data
!9

Visualize
• Kibana
‒ Window into Elastic Stack
‒ Provides Web-based UI to
‒ Manage the stack
‒ Interact with the data
‒ Get data in
‒ And more…
!10

Data Journey
!11
Elasticsearch
Discovery
Visualize
Dashboard
Graph
Kibana
4. Search &
Analyze
Beats
Logstash
1. Genesis
2. Ingest 3. Store

Document
• Document
‒ Serialized JSON Object
‒ Stored in Elasticsearch
‒ Has Unique ID
!12
{
"title": "Fighting Ebola with
Elastic",
"category": "User Stories",
"author": {
"first_name": "Emily",
"last_name": "Mosher"
}
}
title category date author_first_name author_last_name author_company
Fighting Ebola with
Elastic
User
Stories
Emily Mosher
<?xml version="1.0" encoding="UTF-8"?>
<root>
<author>
<first_name>Emily</first_name>
<last_name>Mosher</last_name>
</author>
<category>User Stories</category>
<title>Fighting Ebola with Elastic</title>
</root>
JSON XML
A row in a table

A Simple Example: Spreadsheet
!13
id user age country category
1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 Tom 44 DE B
5 John 40 US B
6 Emma 26 US B

A Simple Example: Elasticsearch
!14
{
"User": "Bill",
"Age": 30,
"Country": "FR",
"Category": "A"
}
Elasticsearch
{
"User": "Tom",
"Age": 44,
"Country": "DE",
"Category": "B"
}
{
"User": "Emma",
"Age": 26,
"Country": "US",
"Category": "A"
}
{
"User": "John",
"Age": 40,
"Country": "US",
"Category": "B"
}
{
"User": "Marie",
"Age": 32,
"Country": "US",
"Category": "B"
}
{
"User": "Claire",
"Age": 32,
"Country": "US",
"Category": "A"
}

Data Categories
• Time Series Data
‒ Event data associated with a moment in time
‒ typically grows rapidly
• Static Data:
‒ relatively slower growth
!15
{
"cuisine": "French",
"ingredients": "Cheese, flour, butter, eggs, milk, nutmeg",
"time_in_min": 50,
"level": "easy"
}
{
"tweet": "Wow Elasticsearch 7.0 seems awesome!",
"hashtags": ["elasticsearch", "kibana"]
"timestamp": September 1st 2017, 07:15:40.035
}
Which category do these
documents belong to?

Elasticsearch Index
• Data Container
‒ Categorical Index
‒ Time Based Index
!16
Elasticsearch
Beats
Logstash
cooking_recipes
tweets-2018-12-24 tweets-2018-12-23 tweets-2018-12-22

Kibana Index Pattern
• Points to one or more Elasticsearch indices
• Tells Kibana which data you want to work with
!17
Index Patterns
Kibana
tweets*
cooking_recipes
Elasticsearch
Indices
cooking_recipes
tweets-2018-12-24 tweets-2018-12-23 tweets-2018-12-22

Messages
!19
Elasticsearch
user_messages
John Smith
Germany
Berlin
130 Followers
#vacation
#dream
{
"message_id": 1,
"user.first_name": "John",
"user.last_name": "Smith",
"user.geo.country": "Germany",
"user.geo.city": "Berlin",
"user.nb_of_followers": 130,
"subjects": "#vacation #dream",
"number_of_subjects": 2,
"likes": 32,
"geo.country": "United Kingdom",
"geo.city": "London"
}

Users
!20
Elasticsearch
user_messages
John Smith
.....
32 likes
John Smith
.....
18 likes
John Smith
.....
123 likes
users
{
"message_id": 41,
"first_name": "John",
"last_name": "Smith",
"geo.country": "Germany",
"geo.city": "Berlin",
"nb_of_followers": 130,
"average_like": 87.45,
"salary": 120000,
"occupation": "Sales"
}

Lab Environment
• Visit Strigo using the link that was shared with you, and log
in if you haven't already done so
• Click on "My Lab" on the left
!22

Lab Environment
• Click on the gear icon next to "My Lab" and select
"Machine Info"
!23

Lab Environment
• Copy the hostname that is shown under "Public DNS"
!24

Lab Environment
• From here you can access lab instructions and guides
‒ you also have them in your .zip file, but it is easier to access and
use the lab instructions from here:
!25

Accessing your Cluster
• Click on the Kibana link:
• Log in
‒ username: training
‒ password: kibana_management
!26

Review - Introduction to Kibana
Kibana Fundamentals
Lesson 1

Summary
• Elasticsearch, Kibana, Logstash, and Beats are
components of the Elastic Stack
• Kibana can be used to analyze, search, interact with and
visualize the data in Elasticsearch
• Kibana can be used to manage the Elastic Stack
• Data is sent as JSON objects into Elasticsearch
• In Kibana, an index pattern can be created to target a
specific set of indices
!28

Quiz
1. What are the four main components of the Elastic Stack?
2. True or False: Data is stored inside Kibana.
3. What would be a suitable index pattern for accessing both
cooking_recipes and cooking_user indices?
4. What kind of dataset the two following documents belong to?
!29
{
"heartbeat": 123,
"timestamp": "Mon, 24 Dec 2018 00:23:28 GMT"
}
{
"first_name": "Bill",
"last_name": "Smith",
"age": 27,
"country": "Mongolia"
}

Kibana Fundamentals
Lesson 1
Lab - Introduction to Kibana

Discover Interface
Kibana Fundamentals
Lesson 2

Overview
• Elasticsearch data types:
‒ numeric
‒ text
‒ date
‒ keywords
‒ ...
• Discover interface
‒ Explore data in Elasticsearch
‒ Slice and Dice (Analyze) Data
!32

Discover Interface
!33
Time picker
Side navigation
Tool bar

Discover Interface
!34
Document table
Histogram
Query bar Index pattern

Search is Everywhere
• Elasticsearch is a search engine
‒ Kibana can be used to search documents in Elasticsearch
• A search is executed by sending a query to Elasticsearch
‒ A query can answer many different types of questions:
‒ who are the users that are called Melissa?
‒ what are the names of the people living in France?
‒ are there any messages about Netflix?
• In Kibana, a search can be executed from the query bar
‒ Kibana supports multiple query languages
!35
*

Querying
• Kibana supports multiple query languages
!36
"Which messages are from John in the US?"
john us
messages-*
1. Define Question
2. Pick Index Pattern
2. Design Query
2. Pick Index Pattern
1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 John 40 DE B
5 John 44 US B
6 Emma 44 US B

Search a Specific Field
• By default, the query below will search all fields for all
values
‒ but being more specific will improve search
• Query above can be made more specific like this
‒ Elasticsearch will only need to search limited fields
!37
john and us
user:john and country:us
What are the messages published by user John from country US?

Boolean Operators
• By default, Kibana uses the or logic
‒ so it matches any documents containing john or us
• Kibana allows you to use the following boolean operators:
‒ and, or, and not
• Now, you can rewrite the query with the and logic
!38
1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 John 40 DE B
5 John 44 US B
6 Emma 44 US B
user:john and country:us

Querying Numeric Fields
• Let's add some complexity to the question:
• Numbers are different than text
‒ instead of exact matches you often have relations:
‒ less than (<)
‒ less than or equal (<=)
‒ greater than (>)
‒ greater than or equal (>=)
• Now, you can rewrite the query as:
!39
What are the messages in which the user is John in the US country
whose age is over 40?
user:john and country:us and age>40

Query "Context"
• Query includes criteria about where to search based on
‒ Distribution in Elasticsearch
‒ Distribution in Time Period
• Make sure to set the correct index pattern and timeframe:
!40
Index Pattern
Time Picker

Demo
!41
Instructor Demo

Review - Discover Interface
Kibana Fundamentals
Lesson 2

Summary
• The discover interface allows you to explore the different
aspects of your data
• The most common mistake in the discover interface is not
checking the index pattern and time picker
• The search bar can be used to search all the data inside
Elasticsearch
• The document table can be customized to display a table of
only selected fields
!43

Quiz
1. What are the first two settings someone should check when
using the discover interface?
2. What are the three different boolean operators?
3. Build the query: "Find the messages from Claire younger
than 30 years old that belong to the category A?"
!44

Kibana Fundamentals
Lesson 2
Lab - Discover Interface

Aggregations
Kibana Fundamentals
Lesson 3

Overview
• Data is often complex and involves many dimensions
• Often, we want summarized insights:
‒ slices based on specific attributes
‒ calculations based on specific attributes
‒ ...
• Spreadsheets might fulfill this using a "pivot table"
• In the Elastic Stack we call the equivalent functionality an
aggregation
• All aggregations are performed at elasticsearch, Kibana just
renders the results
!47

A Simple Example: Spreadsheet
!48
1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 Tom 44 DE B
5 John 40 US B
6 Emma 26 US B

A Simple Example: Elasticsearch
!49
users
Elasticsearch
{
"User": "Bill",
"Age": 30,
"Country": "FR",
"Category": "A"
}
{
"User": "Tom",
"Age": 44,
"Country": "DE",
"Category": "B"
}
{
"User": "Emma",
"Age": 26,
"Country": "US",
"Category": "B"
}
{
"User": "John",
"Age": 40,
"Country": "US",
"Category": "B"
}
{
"User": "Marie",
"Age": 32,
"Country": "US",
"Category": "A"
}
{
"User": "Claire",
"Age": 32,
"Country": "US",
"Category": "A"
}

Metrics Aggregation
• Metric aggregations
‒ Calculates numerical values over a set of documents
‒ similar to how values are summarized in a pivot table for a
specific column
‒ mathematic operation that outputs
‒ a single value (eg., avg, sum, min, max, unique count)
‒ or multiple values (eg., percentiles, percentile_ranks)
!50

A Simple Average Using Pivot Table
!51
1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 Tom 44 DE B
5 John 40 US B
6 Emma 26 US B
AVG of age
34
Rows Values
AVG of age
Pivot table definition Pivot table

A Simple Average Using Aggregations
!52
Elasticsearch
{
"aggregations": {
"avg_of_age": {
"avg": {
"field": "age"
}
}
}
}
"aggregations" : {
"avg_of_age" : {
"value" : 34.0
}
}
{
"User": "Bill",
"Age": 30,
"Country": "FR",
"Category": "A"
}
{
"User": "Tom",
"Age": 44,
"Country": "DE",
"Category": "B"
}
{
"User": "Emma",
"Age": 26,
"Country": "US",
"Category": "B"
}
{
"User": "John",
"Age": 40,
"Country": "US",
"Category": "B"
}
{
"User": "Marie",
"Age": 32,
"Country": "US",
"Category": "A"
}
{
"User": "Claire",
"Age": 32,
"Country": "US",
"Category": "A"
}

Buckets
• Bucket aggregation
‒ A way of slicing data
‒ similar to grouping by values in rows or columns in a pivot
table
‒ Creates buckets
‒ collection of documents that share a common criterion
‒ can have one or more metrics associated with it
‒ Number of documents (doc count) per bucket is default metric
!53

Simple Bucket Using a Pivot Table
!54
1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 Tom 44 DE B
5 John 40 US B
6 Emma 26 US B
category COUNT of id
A 3
B 3
Rows Values
Order ASC by
category
COUNT of id
Pivot table definition Pivot table

Simple Bucket Aggregation
!55
{
"User": "Bill",
"Age": 30,
"Country": "FR",
"Category": "A"
}
{
"User": "Tom",
"Age": 44,
"Country": "DE",
"Category": "B"
}
{
"User": "Emma",
"Age": 26,
"Country": "US",
"Category": "B"
}
{
"User": "John",
"Age": 40,
"Country": "US",
"Category": "B"
}
{
"User": "Marie",
"Age": 32,
"Country": "US",
"Category": "A"
}
{
"User": "Claire",
"Age": 32,
"Country": "US",
"Category": "A"
}
Elasticsearch
{
"aggregations": {
"categories": {
"terms": {
"field": "category"
}
}
}
}
Bucket: A
Count: 3
Bucket: B
Count: 3
"aggregations": {
"categories": {
"buckets": [
{
"key": "A",
"doc_count": 3
},
{
"key": "B",
"doc_count": 3
}
]
}
}

Adding Metrics
!56
1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 Tom 44 DE B
5 John 40 US B
6 Emma 26 US B
category
COUNT
of age
AVG of
age
A 3 31.33
B 3 36.66
Rows Values
Order ASC by
category
COUNT of age
AVG of age

Adding Metrics
!57
"aggregations": {
"categories": {
"terms": {
"field": "category"
},
"aggregations": {
"avg_age_per_category": {
"avg": {
"field": "age"
}
} } } }
"aggregations": {
"categories": {
"buckets": [
{
"key": "A",
"doc_count": 3,
"value": 31.33
}
},
{
"key": "B",
"doc_count": 3,
"value": 36.66
}
} ] } }
{
"User": "Bill",
"Age": 30,
"Country": "FR",
"Category": "A"
}
{
"User": "Tom",
"Age": 44,
"Country": "DE",
"Category": "B"
}
{
"User": "Emma",
"Age": 26,
"Country": "US",
"Category": "B"
}
{
"User": "John",
"Age": 40,
"Country": "US",
"Category": "B"
}
{
"User": "Marie",
"Age": 32,
"Country": "US",
"Category": "A"
}
{
"User": "Claire",
"Age": 32,
"Country": "US",
"Category": "A"
}
Elasticsearch
Bucket: A
Count: 3
Avg of age: 31.33
Bucket: B
Count: 3
Avg of age: 36.66

Nesting Rows/Columns in a Pivot Table
!58
1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 Tom 44 DE B
5 John 40 US B
6 Emma 26 US B
category country
COUNT
of age
AVG of
age
A FR 1 30
US 2 32
B DE 1 44
US 2 33
Rows Values
Order ASC by
category
COUNT of age
Order ASC by
country
AVG of age

Adding Sub-Bucket Aggregation
!59
Elasticsearch
Bucket: A
Count: 3
{
"User": "Bill",
"Age": 30,
"Country": "FR",
"Category": "A"
}
Bucket: FR
Count: 1
Avg of age: 30
{
"User": "Marie",
"Age": 32,
"Country": "US",
"Category": "A"
}
{
"User": "Claire",
"Age": 32,
"Country": "US",
"Category": "A"
}
Bucket: US
Count: 2
Avg of age: 32
Bucket: B
Count: 3
{
"User": "Tom",
"Age": 44,
"Country": "DE",
"Category": "B"
}
{
"User": "Emma",
"Age": 26,
"Country": "US",
"Category": "B"
}
{
"User": "John",
"Age": 40,
"Country": "US",
"Category": "B"
}
Bucket: DE
Count: 1
Avg of age: 44
Bucket: US
Count: 2
Avg of age: 33

Metrics Aggregation
!60
Metrics Aggregation 6
Count of Documents

Bucket Aggregation
!61
DE
FR
US
Metrics
Aggregation
4
3
2
1
0
Bucket Aggregation

Sub-bucket Aggregation
!62
DE
FR
US
Metrics
Aggregation
4
3
2
1
0
Bucket Aggregation
B
A
Sub-Bucket Aggregation

Review - Aggregations
Kibana Fundamentals
Lesson 3

Summary
• Kibana renders visualizations using the results of
Elasticsearch aggregations
• There are two main types of aggregations:
‒ metric
‒ bucket
• Metric aggregations are used to compute numeric values
• Bucket aggregations are used to group data together
!64

Quiz
1. What are the two main types of aggregations?
2. True or False: Aggregations are used by Kibana to render
visualizations.
3. Explain which aggregations are used to build the following
visualization.
!65
B
A
FR
US
DE

Kibana Fundamentals
Lesson 3
Lab - Aggregations

Kibana Search
Module 2
Kibana Search
Kibana Dashboards
Kibana Fundamentals
Kibana Management

Topics
• The Query Bar
• Searching on Text
• Query DSL
• Filters
!68

The Query Bar
Kibana Search
Lesson 1

Improving Search
• Search is central to elastic
‒ In Kibana, ability to search is embedded in almost every page
• However, search is not limited to words we type in a search
bar
‒ or filtering for messages in a specific interval
• Searches may involve:
‒ fuzzy queries
‒ Looking for patterns using regex or wildcard
‒ Searching with bias over certain fields
!70

Wildcard Overview
• In many cases you may want to search for a pattern:
‒ var/lib/elasticsearch/conf/elasticsearch.yml
var/lib/elasticsearch/conf/jvm.options
var/lib/elasticsearch/conf/log4j2.properties
‒ jump, jumps, jumping, jumped
• Wildcard query supports:
‒ * that matches any character sequence (including empty ones)
‒ ? that matches a single character (not available with KQL)
!71

Wildcard Overview
• Both * and ? can be used to search part of a string
• Let's search for every word that start with Mari
• Let's search for every word that starts with Mar?a, in
which ? is any single character
!72
Mari*
Marie
Maria
Marion
Marine
Mari
Mar?a
Marea
Maria
Marya

The Lucene Syntax
• By default the syntax for the query bar is KQL (Kibana
Query Language)
• It can be changed in the search bar to the Lucene syntax if
needed
• The advantage of the Lucene query is that:
‒ it offers more features such as regex query or fuzzy query
‒ people familiar with Elasticsearch will be able to use it easily
• A third language called the query DSL can be used (this will
be covered later in this training)
!74
Search KQL
Search Lucene

Fuzzy Search
• Users expect search applications to adapt to spelling errors
• We can use an approach known as edit distance:
‒ edit distance is a function that counts the number of edits
required to change one word into another
‒ Damereau-Levenshtein distance is a popular method
!75
“Mario”
o e
Marie
Edit distance = 1
“Eifele”
e
“Eiffel”
Edit distance = 2
f

Fuzzy Search
• The edit distance can be defined per query term
‒ allowed values are 0 (default), 1, 2, and auto
‒ auto will define the fuzziness based on the length of the word
!76
Mario~1
Eifele~2 Eiffel
Marie
...
...
Edit distance
value
Fuzziness
Searched
word
Mario~auto Marie
...

Fuzzy Search
• Be aware that fuzzy search is expensive
‒ it will not be as fast as regular searches
‒ and it can crash your cluster
• Great for incident analysis
• More computationally expensive than regular search
!77

Regexp Query
• The regexp query greatly enhances our ability to search for
patterns
‒ syntax is based on the Lucene regular expression engine
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html#regexp-syntax
• For example, you can use it to find all the documents that
contain a phone number in France:
‒ starts with +33 followed by 9 numbers
!78
/.*+33[0-9]{9}.*/
my phone number: +33235151151
Here: +33235151151 call me later
...

Searching for Patterns is Expensive
• Both wildcard queries and regexp queries are expensive
‒ they will not be as fast as regular searches
‒ and they can crash your cluster
• Great for incident analysis
• Not so great for common queries
‒ Always avoid leading patterns (eg: *aria)
!79

Boosting
• When searching on multiple fields, finding something in
certain fields may mean more to you than others:
‒ for example, an email title is probably more relevant than the
body
• In the search bar, it is possible to reflect this by boosting
specific fields:
‒ which impacts the order of the returned documents
!80
subjects:instafood OR first_name:john^2
Searching on the subjects
and first_name fields
Boosting the score by 2
on the field first_name

Review - The Query Bar
Kibana Search
Lesson 1

Summary
• Fuzzy, Wildcard, and Regular Expression queries are
powerful and can help you find documents with partial
information
• However, they are expensive and should be used with care
• Boosting allows you to influence the order that documents
are returned
• Kibana provides multiple languages to search over your
data:
‒ KQL
‒ Lucene
!82

Quiz
1. True or False: Fuzzy, Wildcard, and Regular Expression
queries are powerful and cheap.
2. True or False: Kuery is enabled when you turn on the
autocomplete feature.
3. Explain the following query:
!83
subjects:insta* AND user:maria~auto

Kibana Search
Lesson 1
Lab - The Query Bar

Searching on Text
Kibana Search
Lesson 2

Text Subtleties
• Search calls for different behavior based on different types
of text fields:
• Let's take the following example:
• Both are text, but
‒ a search for "paris" should still find documents that contain
"Paris"
‒ a search for "B6IAWRLOLB" should not find documents that
contain "B6IAWRLOLb"
• Why?
!86
Comment: Wow! I love this city, Paris is beautiful!
User ID: B6IAWRLOLb

Full Text Search vs Exact Match
• Text search can be:
‒ full text search
‒ a search for "paris" should still find documents that contain "Paris"
‒ Requires normalization (eg., Paris => paris)
‒ exact match
‒ a search for "B6IAWRLOLB" should not find documents that contain
"B6IAWRLOLb"
‒ No normalization (as is)
• Let's dive more into those two use cases
!87

Text Analysis
• Text Analysis
‒ Elasticsearch Analyzer performs text analysis
‒ Customizable
‒ lower casing, removing punctuation, tokenization, …
• Let's index documents in Elasticsearch:
!88
Wow! I love this city,
Paris is beautiful!
Analysis
Tokens ID
wow 1
i 1, 2
love 1, 2
this 1
city 1
paris 1, 2, 3
is 1
so 1
beautiful 1
1
I love Paris!
2
Paris
3

Query Analysis
• When querying an analyzed field, by default, the query will
also be analyzed
‒ and then compared to the tokens of the indexed documents
!89
The city of Paris Analysis
Tokens ID
wow 1
i 1, 2
love 1, 2
this 1
city 1
paris 1, 2
is 1
so 1
beautiful 1
the city of paris
1 2
Query
Response

Exact Match
• For exact match query, text will not be analyzed
‒ no normalization
• Let's index documents in Elasticsearch:
!90
Wow! I love this city,
Paris is beautiful!
Tokens ID
Wow! I love this
city, Paris is
beautiful!
1
I love Paris! 2
Paris 3
1
I love Paris!
2
Paris
3

Exact Match Query
• When querying a unanalyzed field, by default, the query will
not be analyzed either
‒ but directly compared to tokens of indexed documents
• It should be used for strict values instead of sentences
!91
Tokens ID
Wow! I love this
city, Paris is
beautiful!
1
I love Paris! 2
Paris 3
Paris
paris
3

Index Patterns
• In Kibana, Index pattern provides a summary of fields in
elasticsearch index
‒ Along with properties that indicate whether they are suitable for
‒ Certain Visualization
‒ Exact match queries
‒ Full text search
!92
EXACT MATCH FULL TEXT SEARCH
AGGREGATABLE ✓
SEARCHABLE ✓ ✓

Review - Searching on Text
Kibana Search
Lesson 2

Summary
• Text data has two main use cases: full-text search and
exact match
• In full-text search, data needs to be normalized
• In exact match, data is not normalized
• Queries may be analyzed or not depending on our query
and the data type
• In Kibana, the index patterns view shows this information
for each of the fields
!94

Quiz
1. What are the two main use cases for text data?
2. True or False: When querying, by default, the query will be
analyzed if the field is setup as an exact match.
3. In Kibana, where can you check if a field should be used for
search or for aggregations?
!95

Kibana Search
Lesson 2
Review - Searching on Text

Query DSL
Kibana Search
Lesson 3

Search with Query DSL
• So far, we discussed two different query syntax that Kibana
supports:
‒ Lucene syntax
‒ KQL (has auto-completion)
• Another syntax uses Query DSL:
‒ it is mostly used by developers to query Elasticsearch directly
‒ However, DSL offers a way to fine tune and add flexibility to
Kibana searches
!98

The Query DSL Syntax
• Query DSL syntax is based on JSON to write complex
queries:
‒ JSON (Java Script Object Notation)
‒ makes it easy to read and understand queries
‒ is comprised of a set of key/value pairs
!99
GET users*/_search
{
"query": {
"TYPE_OF_QUERY": {
...
}
}
}
"Index Pattern"
GET users*/_search
{
"query": {
"match": {
"subjects": {
"query": "#dogs"
}
}
}
}
Values
Keys

Match Query
• Imagine you want to search for "food", "dogs" or "chocolate"
in the subjects field
• Using the Lucene syntax, you would write the following:
• Using the query DSL syntax, you would write the following:
!100
subjects:(food dogs chocolate)
GET user_messages*/_search
{
"query":{
"match": {
"subjects": {
"query": "food dogs chocolate"
}
}
}
}

Match Query Operator
• By default the match query applies the OR operator
‒ food OR dogs OR chocolate
• You can change that behavior by explicitly defining an
operator
!101
{
"query":{
"match": {
"subjects": {
"query": "food dogs chocolate",
"operator": "and"
}
}
}
}

Minimum Should Match
• Searching for "food", "dogs" or "chocolate" may be too
permissive and give too many results
• Elasticsearch allows you to specify the minimum number of
terms that should match
‒ for example, at least 2 words out of 3 should match
‒ (food AND dogs) OR (food AND chocolate) OR (dogs AND chocolate)
!102
{
"query":{
"match": {
"subjects": {
"query": "food dogs chocolate",
"minimum_should_match": 2
}
}
}
}

Going Deeper
• The query DSL syntax may seem complex at first but is
very powerful
• There are many types of queries:
‒ match
‒ match_phrase
‒ range
‒ ...
• The documentation will help you build queries:
‒ https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html
• Engineer I and Engineer II trainings cover it in depth
!103

Demo
!104
Instructor Demo

Review - Query DSL
Kibana Search
Lesson 3

Summary
• The Query DSL is more verbose
‒ but offers more control
• The match query is a simple query to find all documents
that match query content
• The match query uses an OR operator by default, but it can
be easily changed to an AND operator
• It is possible to match a few out of many terms in a match
query by setting the minimum_should_match parameter
with in a "match" query
!106

Quiz
1. True or False: The Query DSL allows you to use every
search feature in Elasticsearch.
2. What is the default operator of the match query?
3. True or False: By default, the match query should match a
minimum of two terms.
!107

Kibana Search
Lesson 3
Lab - Query DSL

Filters
Kibana Search
Lesson 4

Query Bar Limitations
• Imagine you have multiple search criteria:
‒ age>30
‒ subjects:art
‒ subjects:fashion
‒ subjects:summer
‒ first_name:Claire
• Often times, you may want the ability to turn on/off these
criteria individually
• Using the query bar will require a lot of typing and deletions
to accomplish this
• Filters can come in very handy to do just that…
!110

Filters Overview
• A filter is a graphical control equivalent to a search criteria
• Once defined a filter can be:
‒ enabled/disabled
‒ pinned
‒ negated/positivized
‒ dropped
‒ edited
!111

Filters Customization
• Internally filters are transformed to a query DSL
• There are two ways of customizing a filter:
‒ adding a label to the filter to quickly identify it
‒ redefine the way the filter behaves by editing the query DSL
!112

Filter in Discover
• Filters can be used to quickly filter and explore data:
1. create multiple relevant filters
2. type a query
3. disable/Enable filter to explore data
!113

Filter Across Kibana
• Filters can be used to navigate across multiple Kibana
interfaces
1. create filters in discover
2. pin interesting filters
3. navigate to Visualize
4. create a Visualization
!114

Filter and Query Bar
• Filters and the query bar are complementary
• It is possible to type multiple search criteria in the query bar,
however when executed all criteria will be applied at the
same time
• Filters provides the ability to quickly enable/disable criteria
individually using mouse clicks
• In the background, filters gets translated to queries and
they will have similar performance
!115

Internals
• Internally the Lucene query inside the query bar will be
transformed into a query_string:
‒ https://www.elastic.co/guide/en/elasticsearch/reference/current/
query-dsl-query-string-query.html
• Depending of the created filter, different kinds of queries will
be generated:
‒ range: https://www.elastic.co/guide/en/elasticsearch/reference/
current/query-dsl-range-query.html
‒ match_phrase: https://www.elastic.co/guide/en/elasticsearch/
reference/current/query-dsl-match-query-phrase.html
‒ ...
• When using KQL they will be transformed into regular Query DSL
language (same as filters).
• Both filter and search will be pass as a must clause of a bool query
!116

Demo
!117
Instructor Demo

Review - Filters
Kibana Search
Lesson 4

Summary
• Kibana filters provide an easy way to explore data by
‒ Enabling and disabling them
‒ Pinning and having them follow to different parts of Kibana
• You can customize filters using:
‒ Pre-defined settings
‒ Editing Generated Query DSL
• Kibana filters and the query bar are complementary
!119

Quiz
1. True or False: Kibana only allows a single filter at a time.
2. Name three actions that you can perform on a filter.
3. True or False: You should use either Kibana filters or the
query bar.
!120

Kibana Search
Lesson 4
Lab - Filters

Module 3
Kibana Search
Kibana Dashboards
Kibana Fundamentals
Kibana Management

Topics
• Visualization Refresher
• Pipeline Aggregations
• More Types of Visualization
• Improving Visualizations
!123

Visualizations Refresher
Lesson 1

A Small Refresher ...
• A visualization is created out of aggregations
• There are different kind of aggregations:
‒ Metrics (to compute a value)
‒ Bucket (to group documents into categories)
‒ Pipeline (this is going to be covered later in this training)
‒ Matrix (to produce a matrix based result)
• Metrics aggregations are used to quantify things (how big
should be a bar, or a slice)
• Buckets aggregation are used to define how the data are
going to be sliced (creating the slices or the bars)
• Let's do a lab to learn more about the aggregations...
!125

Lesson 1
Lab - Visualizations Refresher

Pipeline Aggregations
Lesson 2

A Simple Problem
• New users participate every month:
‒ How would you build the cumulative sum of this chart?
!128
Number
of
users
100
0
200
April
Mai
June
July
August
September
October
November
December
January

A Simple Solution
• Stacking different buckets will solve the issue...
!129
Number
of
users
100
0
200
300
400
April
Mai
June
July
August
September
October
November
December
January
500
0
1000
1500
2000
Cumulative
sum
of
users

Introduction to Pipeline Aggregation
• So far, we have seen aggregations that work on search
results
• A pipeline aggregation, will actually works on the output
generated by another aggregation
• There are multiple pipeline aggregations available:
‒ avg, max, min, sum, cumulative sum, derivative, moving avg,
serial diff
• Let's dive into some of the most complex pipeline
aggregations…
!130

Derivative
• The derivative is used to compare one bucket with the
previous one:
!131

Derivative
• No value can be computed for the first bucket as it requires
at least 2 buckets
!132

Derivative
!133

Derivative
• Derivative helps us understand how values change
‒ the longer the bar is, the greater the difference from the previous
value
!134

Serial Differencing Aggregation
• The serial differencing aggregation will compute the
difference between two buckets separated by a defined lag:
‒ A lag of 1 (default) makes this aggregation equivalent to a
derivative
!135

Serial Differencing Aggregation
• A pattern seems to repeat every 9th bucket:
‒ Let’s apply serial differencing with a lag of 9
‒ Instead of every bucket, we calculate difference every 9th bucket
!136

The Moving Average
• Moving average can be used to smooth variations in time
series data in order to highlight general trends
!137

Window Based
• A window will be defined on top of the buckets
‒ Buckets falling into this window will be averaged
‒ It is possible to define the size of the window
!138

Window Based
• The function then slides the window to compute average
!139

Window Based
• Until all buckets are processed
!140

Window Based
• The end result is a smoother time series:
!141

Advanced JSON input
• The Visualize UI doesn't provide access to all the
parameters of an aggregation (some aggregations include a
lot of parameters!)
• The advanced JSON input can be used to customized
certain parameters:
‒ changing the window size for a moving average
‒ changing the lag in a serial differencing aggregation
‒ ...
• To customize aggregation:
‒ refer documentation pages for that aggregation
‒ Obtain parameters to tweak aggregation
‒ Add parameters to the advanced JSON input
!142

Demo
!143
Instructor Demo

Review - Pipeline Aggregations
Lesson 2

Summary
• Pipeline aggregations work on the output of another
aggregation instead of search results
• Default settings in serial differencing aggregation makes it
equivalent to the derivative aggregation
• Pipeline aggregations have a lot of parameters that can be
added using Advanced JSON input
!145

Quiz
1. What parameter should be used to increase the window
interval of a moving average?
2. True or False Pipeline aggregations work directly on search
results
3. What is the difference between the derivative and the serial
differencing aggregation?
!146

Lesson 2
Lab - Pipeline Aggregations

More Types of Visualization
Lesson 3

Pivot Table in Kibana
• Aggregations and pivot tables are two concepts that are
very similar
• By using the data table visualization it is possible to have a
"pivot table" like structure in Kibana
• What aggregations are needed in order to generate the
following table?
!149
Category Country Count Average(age)
Category A US 2 32
Category A FR 1 30
Category B US 2 33
Category B DE 1 44

Pivot Table
• We need the following to create the table:
‒ a bucket aggregation of type term on the category field first
‒ a sub-bucket aggregation of type term on the country field
‒ a metric aggregation of type count (default metric aggregation)
on all generated buckets
‒ another metric aggregation of type average on the age field on
all the generated buckets
!150

Table Visualization
• By default data table in Kibana will have only one row
‒ Showing a count of all documents
!151
Count
6

Table visualization
• The row can be split by applying the first term
aggregation on the field category
!152
Category Count
Category A 3
Category B 3

Table visualization
• Resulting rows can then be split again by applying
another term aggregation on the country field
!153
Category Country Count
Category A US 2
Category A FR 1
Category B US 2
Category B DE 1

Table visualization
• A column can then be added to the table by adding the
average metric aggregation on the age field:
!154
Category Country Count Average(age)
Category A US 2 32
Category A FR 1 30
Category B US 2 33
Category B DE 1 44
Total 6 139

Customizing Data Table
• Every row in a table is the result of a bucket aggregation
• The number of rows displayed in the table can be
customized using the options tab
• Tables can span multiple pages and you can paginate to
subsequent pages using the link at the bottom of the table
• It is possible to apply the metric aggregations at the parent
level using the options tab
• A grand total of all the metrics can also be computed
!155

Heat Map Overview
!156
DE US FR
Category A
Category B
Metric Aggregation
Bucket Aggregations

Heatmap Example
!157
0k-10k
10k-20k
20k-30k
30k-40k
40k-50k
50k-60k
60k-70k
70k-80k
Haiti Jordan Mayotte Venezuela Lebanon Botswana
Countries and download size by response time

Tag Cloud Overview
• Tag cloud is way to represent words in a dataset visually
• It supports only two bucket aggregations:
‒ terms aggregation
‒ significant term aggregation
!158

Tag Cloud
• Every word is the result of the bucket aggregation
• The size of a word is a function of the result of the metric
aggregation (default to count)
!159
Marie
Tom
John Emma
Bill

Visualizing Geo Points
• The Coordinate map allows visualization of geo points on a
map
• A grid is generated on top of a Mercator map
• Granularity of the grid will depend of zoom level
‒ user’s zoom action will yield higher precision on map
• Visualized points need to be of type "geo_point" in
Elasticsearch
!160

Low Precision
!161

Higher Precision
!162

Region Map Overview
‒ Unlike Coordinate maps that allows visualization of geo points,
Region map helps visualize regions:
‒ country
‒ states
‒ provinces
• Kibana ships with multiple vectors maps
‒ https://maps.elastic.co/v2/index.html#
• Elastic Map Service has zoom levels up to 10
‒ 18 with basic license
• It is also possible to use other base maps
‒ https://www.elastic.co/blog/custom-basemaps-for-region-and-
coordinate-maps-in-kibana
!163

Region Map Creation
• Data should include a join field
‒ Terms in document to connect to properties in vector map
‒ For example
‒ 2 letters country code: FR, US, DE, NL
‒ 3 letters country code: FRA, USA, DEU, NLD
‒ the country name: France, United States, Germany, Netherlands
‒ Defined under options tab
• Run terms aggregation on the join field
• Define metric aggregation
‒ Color & intensity of region represents metric value
!164

Region Map
!165

Vega
• Vega is an open source visualization grammar
• Vega is available from Kibana 6.2+
‒ Kibana 6.2.4 supports Vega 3.2.1 and Vega-Lite 2.3.1
‒ Kibana 6.3.0 supports Vega 3.3.1 and Vega-Lite 2.4.0
• Users can build custom, interactive visualizations that can
be integrated into their Kibana dashboards.
• Kibana’s Vega is a very technical tool which requires a
deep understanding of both the Vega language and
Elasticsearch: Specialization course about Vega
!166

Demo
!167
Instructor Demo

Review - More Types of Visualization
Lesson 3

Summary
• Kibana provides a lot of ways to visually represent data.
• Elasticsearch aggregation concepts are central to building
Kibana visualization
• In Heat Maps, bucket aggregations on X and Y axis
produces the cells of the matrix and the result of metric
aggregation determines color or intensity of individual cells
• To visualize data on maps, you can use coordinate map
(documents should have a field with latitude and longitude)
or region map (documents should have a field containing a
region)
!169

Quiz
1. What is the maximum zoom level in Elastic Map Service?
2. How would you create the following table:
!170
Name Count Average Like
Smith 321 25.4
Goodwill 219 193.4
De Bourraine 200 149.4
Schwartz 143 123.8

Lesson 3
Lab - More Types of Visualization

Improving Visualizations
Lesson 4

Comparing Multiple Metrics
!173
DE
FR
US
Bucket Aggregation
Metrics
Aggregation
4
3
2
1
0
?
Document
count
40
30
20
10
0
Average
age

Multi Metrics Visualization
• Multiple metrics can be displayed side by side in a
visualization
• Having multiple metrics can make the visualization hard to
read for the following reasons:
- the metrics have different scale
- the metrics are stacked (or not, depending of the purpose of the
visualization)
- the metrics have same representations when comparing different
variables (bars, lines, area)
- poor color choice
• Each metric can have it’s own style, axes, color, chart
type, ....
!174

Multi Metrics Visualization
• Metric Aggregation can be customized with it’s own style:
- colors
- axes
- properties
• Each metric can be displayed differently:
- lines
- bars
- areas
!175

Multi Metric Visualization
!176
Two different Y axes
Different chart
types

Bubble Chart
• Bubble chart can be built by defining a new metric:
1. define X-axis
2. define Y-axis
3. define dot size
4. change visualization to display a line
5. uncheck "show line"
!177

Bubble Chart
!178

Multi Charts
• Sometimes, multiple buckets or sub-buckets on a single
visualization may not be suitable:
‒ Buckets or sub-buckets may not be related
‒ Too many metrics on the same visualization can make it busy
• In Kibana it is possible to display multiple charts in the
same visualization by using split chart
!179

Single Chart
!180
DE
FR
US
4
3
2
1
0
B
A Document
count
40
30
20
10
0
Average
age
Count Average

Multi Charts
!181
FR
US
2
1
0
Document
count
40
20
0
Average
age
DE
US
2
1
0
Document
count
40
20
0
Average
age
Category A
Category B

Demo
!182
Instructor Demo

Review - Improving Visualizations
Lesson 4

Summary
• Multiple metrics can be displayed in a single visualization
• Every metric can be customized, with a dedicated axis,
different color, style, ...
• It is possible to create multiple charts inside a single
visualization by using a bucket aggregation to divide the
data
!184

Quiz
1. What is the minimum number of metrics required for a bubble
chart?
2. True or False: Splitting a chart relies on the same concept as
a bucket aggregation
3. True or False: It is possible to display 2 metrics that have
very different scales on the same visualization
!185

Lesson 4
Lab - Improving Visualizations

Kibana Dashboards
Module 4
Kibana Search
Kibana Dashboards
Kibana Fundamentals
Kibana Management

Topics
• Introduction to Dashboards
• Markdown and User Input
• Anomaly Hunt
!188

Introduction to Dashboards
Kibana Dashboards
Lesson 1

Limitations
• Visualizations are interactive, it is possible:
‒ to filter out some values
‒ apply searches
‒ select a time window
• However, there is still a lot that we cannot do with just
visualizations:
‒ compare different visualizations for the same time window
‒ compare raw documents alongside other visualizations
‒ drill down (filter) in one visualization (eg: map zooming) and
analyze impact in another visualization
!190

Dashboards
• We can do all that and more with Dashboards:
!191

Dashboards Overview
• Collection of Visualizations and Saved Searches
• Requires at least one visualization or saved search
• To create a new dashboard
‒ under Dashboard, Click "Create New Dashboard"
‒ click "Add" to add visualizations and searches
‒ resize, Rearrange, Reorder elements
‒ save dashboard
!192

Filters and Dashboards
• One of the main advantages of dashboards is the ability to
dynamically drill down on data
• Different actions on visualizations dynamically creates
different filters:
‒ clicking on a bucket (bar, pie-slice, etc.,)
‒ zooming on map (double-click, rectangular select, etc.,)
‒ using input control (this will be covered later on)
‒ using search bar query
!193

Sharing Dashboard
• Once a dashboard is saved, you can share it
‒ using a permalink
‒ using an iframe (embedded)
‒ generating a report
‒ users will need Kibana access to view an embedded dashboard
!194

Inline Frame (IFrame) and Permalink
• IFrames allow Kibana dashboard (or visualization) to be
embedded in an HTML document
‒ Copying an IFrame will provide html code wrapped in iframe tag
• Permalinks provides hyperlink that points directly to a
Kibana dashboard or visualization
‒ URL may be quite long since it includes state information
‒ URL Encoded
‒ URL Shortening available as well
• Both Iframe and Permalink can generate link for either:
‒ Snapshot
‒ Saved object
!195

Snapshot of a Dashboard
• To summarize the difference between snapshot and saved
objects:
‒ Snapshot:
‒ dashboard state is frozen at create time and will not change even if
underlying data changes
‒ Saved Object:
‒ dashboard will render visualizations real-time based on current data
!196

Kibana Access
• In order to access shared dashboard
‒ You need Kibana access
‒ Appropriate user permissions
‒ Read-only
‒ Read, write and delete
• Elastic security can help manage
‒ Users
‒ Roles
‒ Permissions
!197

Iframe and Kibana Security
• When using iFrame to access Secured instance of Kibana
‒ Multiple sign-ins may be required
‒ Application embedding iFrame
‒ Kibana
• To avoid this inconvenience you can:
‒ define and use a kibana_dashboard_only_user
‒ use reverse proxy to pass authentication for the user
!198

Canvas
• Canvas is a rich live infographic system
• By using Canvas you will be able to create:
‒ Dashboards
‒ Reports
‒ Anything to represent data
• Canvas is Beta since 6.5
!199

Canvas
!200

Demo
!201
Instructor Demo

Review - Introduction to Dashboards
Kibana Dashboards
Lesson 1

Summary
• Dashboards group visualizations and searches in a single
place to facilitate analysis
• Dashboards can be shared in multiple ways with other
users
• Sharing a dashboard may need security considerations:
‒ Does the person have access to Kibana?
‒ Does the person have access to shared data?
!203

Quiz
1. True or False One of the limitations of visualization is their
inability to have filters
2. What is the difference between sharing a saved dashboard
and a snapshot of a dashboard?
3. What should someone be careful about when sharing a
dashboard?
!204

Kibana Dashboards
Lesson 1
Lab - Introduction to Dashboards

Markdown and User Input
Kibana Dashboards
Lesson 2

Markdown Overview
• A simple markup language
‒ Supports text annotations and basic formatting
‒ define links
‒ add images
‒ define titles
‒ …
‒ Not designed for advanced formatting
!207

Markdown Example
!208
# This is a big title
## ... and a smaller one
-----
* A list can be useful
* ... to display links to _dashboards_
* ... for __instance__
This is a big title
... and a smaller one
_________________
• A list can be useful
• ... to display links to dashboards
• ... for instance

Markdown Visualization
• Kibana supports Markdown widget
‒ Provide supporting text for dashboard and elements
‒ Embed values dynamically from data
‒ add links to other dashboards (that may be related)
‒ ...
!209

Dashboards and Filters
• Just as in Discover, you can add a filter in dashboards
• However, controls visualization provides an intuitive way
to let analysts filter for specific fields
• With controls visualization, you can:
‒ add a slider for numeric values
‒ add an options list to pick keywords
!210

Demo
!211
Instructor Demo

Review - Markdown and User Input
Kibana Dashboards
Lesson 2

Summary
• There is a dedicated markdown visualization to add textual
information to your dashboard
• Inside a markdown visualization it is possible to add links
that point to other source of information
• Controls visualization can be used to generate filters
directly inside the dashboards
!213

Quiz
1. True or False It is possible to have dynamic text in a
markdown visualization
2. Which text will be the biggest:
1. # Title1
2. ## Title 2
3. True or False The filter generated by the controls
visualization are different than the one that you can generate
manually
!214

Kibana Dashboards
Lesson 2
Lab - Markdown and User Input

Anomaly Hunt
Kibana Dashboards
Lesson 3

Demo
!217
Instructor Demo

Review - Anomaly Hunt
Kibana Dashboards
Lesson 3

Summary
• To hunt for an anomaly, use the different visualizations
present inside the dashboard to dive into a specific anomaly
• Once an anomaly is detected, it is possible to remove the
anomaly by creating the corresponding filter
• A filter can be pinned in order to navigate through multiple
dashboards while hunting an anomaly
!219

Quiz
1. True or False: An anomaly is always shaped by a high value
on a visualization
2. How can an anomaly be removed from the visualizations?
3. Is that always advantageous to remove anomalies in a
dashboard?
!220

Kibana Dashboards
Lesson 3
Lab - Anomaly Hunt

Module 5
Kibana Search
Kibana Dashboards
Kibana Fundamentals
Kibana Management

Topics
• Visual Builder for Time Series
• Visual Builder Aggregations
• Visual Builder and Other Visualizations
!223

Visual Builder for Time Series
Lesson 1

Introduction
• Time Series Visual Builder
‒ Combines a wide range of aggregations (including pipeline
aggregations) to analyze time series data in a meaningful way
‒ Supports many customization options in how data is visualized in
chart
‒ background colors
‒ axis
‒ ...
‒ Supports combining multiple index patterns in the same
visualization
‒ Supports annotations
!225

More Aggregations and Flexibility
• Visual Builder
‒ Supports a wide range of aggregations:
‒ math (to apply mathematic operations to the data)
‒ static values
‒ overall maximum and minimum
‒ and many more ....
‒ Offers more flexibility
‒ shifting the time series
‒ cloning series
‒ styling
‒ multiple index patterns
!226

Annotations
• Using Visual Builder for time series, it is possible to add
annotations to highlight important events in your time series
• Every annotation comes with a tooltip that can be
customized using a templating language called Mustache
• Mustache has a fairly syntax:
!227
Warning {{error_type}} found!

Timelion
• Timelion is another powerful time series data visualizer in
Kibana
‒ Geared towards users who prefer writing code
‒ Driven by a simple expression language
‒ Uses "." chained functions
‒ Query bar supports code completion
!228

Demo
!229
Instructor Demo

Review - Visual Builder for Time
Series
Lesson 1

Summary
• Time Series Visual Builder is a powerful tool that offers a lot
of flexibility when working with aggregations
• TSVB offers many more options compared to regular
visualizations in
‒ Styling options
‒ Crossing index patterns
‒ Metrics manipulation
‒ …
!231

Quiz
1. Give 3 advantages that Time Series Visual Builder has over
other visualizations
2. True or False: It is possible to define static value using Time
Series Visual Builder
3. True or False: It is possible to define an offset on a time
series to align patterns
!232

Lesson 1
Lab - Visual Builder for Time Series

Visual Builder Aggregations
Lesson 2

Static Value
• Visual Builder support a lot of additional aggregations
• Using the pre-build visualizations it was not possible to
display a static value but by using the static value in Visual
Builder is is possible to do so:
‒ To represent a value that need to be reached
‒ To represent a value that other metrics should not be reach or go
above
!235

Math
• The math aggregation in Visual Builder allows the execution
of mathematical operations
• When creating the math aggregation multiple information
are required:
‒ variables
‒ expression
• The variables are defined using other aggregations
• The expression is defined using a TinyMath expression
!236

Filter Ratio
• The filter ratio aggregation is used when there is a need to
compute a ration between two set of values
• The set of values is defined by the creation of a filter for
both the numerator and denominator
• The filter is expecting a query in the Lucene syntax
!237

Demo
!238
Instructor Demo

Review - Visual Builder Aggregations
Lesson 2

Summary
• Visual Builder give you access more aggregations
• TinyMath syntax can be used to define the mathematical
expression
• The filter ratio aggregation can be used to do mathematical
operation between subset of data belonging to the same
index pattern
!240

Quiz
1. What is the expression language used to generate math
expression?
2. True or False: It is possible to combine a static value
aggregation and a math aggregation?
3. True or False: It is possible to use KQL to define the ratios in
a filter aggregation
!241

Lesson 2
Lab - Visual Builder Aggregations

Visual Builder and Other
Visualizations
Lesson 3

Other Visualizations
• Visual Builder is not limited to visualizing time series
• It can also be used for
‒ metrics
‒ gauges
‒ top N
‒ markdown
‒ tables
• Those visualizations allow the addition of conditional to
change the color of the visualization based on certain
criterion
• Visualizations do not work on the whole time series but only
on the last bucket of the time series
!244

Metric
• It is possible to visualize up to two metrics using the metric
Visual Builder visualization
• Multiple visualizations can be generated if the documents
are split using for instance a term aggregation
• Colors can be defined based on thresholds:
‒ for the metric itself
‒ for the background color
!245

Top N
• Top N visualization is representing data in a vertical bar
chart
• The bars can be generated using:
‒ the term aggregation
‒ by defining multiple series
• Conditions can be defined to change the color of the
background dynamically based on thresholds
• Color of the background can be customized
!246

Markdown
• The Visual Markdown visualization can be used to:
‒ introduce dynamic elements in textual data
‒ customize the style of the visualization
• Introduction of dynamic elements can be done using the
mustache syntax
• Customization of the style will be done using Cascading
spread sheet (CSS)
!247

Table
• The table visualization that Visual Builder is providing is
fairly similar to the prebuilt one
• It provide a way of customizing the color of the metric
displayed based on thresholds at the aggregation level
(column level)
• There is more aggregation available than using the prebuilt
visualization
!248

Demo
!249
Instructor Demo

Review - Visual Builder and Other
Visualizations
Lesson 3

Summary
• Visual Builder visualization can be used to create 6 different
type of visualization
• Except for the Time Series visualization all visualization are
only displaying data from the last bucket
• For every visualization, thresholds can be defined to
change the color of the visualization
• Markdown can be customized using CSS
!251

Quiz
1. What syntax is going to be used to add dynamic elements in
the markdown visualization?
2. True or False: All the visualization in Visual Builder display
data from the whole data set define in the time picker
3. True or False: The color threshold for the table visualization
are applied at the table level
!252

Lesson 3
Lab - Visual Builder and Other
Visualizations

Kibana Management
Module 6
Kibana Search
Kibana Dashboards
Kibana Fundamentals
Kibana Management

Topics
• Advanced Settings
• Reporting and Saved Objects
• Security and Spaces
!255

Advanced Settings
Kibana Management
Lesson 1

Scripted Fields
• Scripted fields compute values on the fly based on a script
‒ Value computed at query time and not indexed
‒ Can be very resource intensive and can impact Kibana’s performance
‒ there is no validation
‒ Buggy scripts will generate exceptions
‒ Can take "Painless" or "Lucene Expressions" for scripts
modules-scripting-painless.html
modules-scripting-expression.html
!257

Scripted Fields
• Create a Scripted field
‒ From Management > Index Patterns
‒ You can also view all previously created scripted fields
‒ Once created, can be used like a regular field
‒ For building visualizations
‒ For searching (Supports only KQL)
!258

Quick Range
• Time picker defaults to last 15 minutes
• If this default doesn’t suite your use case you can change it
in under advanced settings:
‒ Use timepicker:timeDefaults
• It is as well possible to define different shortcut under:
‒ quick tab
‒ absolute tab
‒ relative tab
‒ Provide appropriate values for from and to parameters
!259

Timezone
• If your timestamp field contains timezone information
‒ Kibana will adjust time to use browser timezone by default
• You can override this behavior
‒ Use dateFormat:tz parameter
!260
ES DOC TIME (UTC) dateformat:tz KIBANA
2019-04-26T13:04:37.742Z browser (AMERICA/New_York) Apr 26, 2019 @ 09:04:37.742
2019-04-26T13:04:37.742Z EUROPE/Paris Apr 26, 2019 @ 15:04:37.742

Format
• Numbers can be formatted differently depending of what
they represent:
‒ bytes
‒ duration
‒ percentage
‒ color
‒ ...
• It can be defined per field under index pattern
• Or globally for all fields under advanced settings
!261

Locale Number
• Number formatting differs by region
• Let’s take 30,000 for instance:
‒ in France comma is used to represent decimal and hence this
number will be read as thirty
‒ in English speaking counties this number will be read as thirty
thousand
• format:number:defaultLocale allows defining default locale for numbers
• Based on this setting, the number Three Thousand point zero one will be
displayed
‒ As 3 000,01 when set to fr
‒ As 3 000.01 when set to en
!262

Displayed Documents
• By default only first 500 documents are displayed in
discover interface
• It is possible to override this default by:
‒ Updating discover:sampleSize parameter
‒ Displaying too many document may negatively impact user
experience
• Meta-fields can be removed from the discover interface:
‒ keep only the _source in the metafields
‒ the _source field holds the original document that was handed to
Elasticsearch for storage
!263

Search
• Kibana Query Bar supports wildcard queries
‒ It’s expensive
‒ Especially Leading wildcards
‒ Disable for Lucene queries using query:queryString:options
‒ Disable for autocompletion using query:allowLeadingWildcards
• If the query is not field specific
‒ Kibana will search on all fields
‒ query:queryString:options can be used to change default field
!264

Instructor Demo
Demo
!265

Review - Advanced Settings
Kibana Management
Lesson 1

Summary
• Values in Kibana Time picker can be customized by adding
pre-defined time intervals
• Scripted field is a way to create a new field that’s not part of
your data:
‒ Can be very expensive
‒ Work with your data admins to pre-compute and store frequently
used scripted fields
• Numbers can be formatted to reflect local representation
!267

Quiz
1. True or False: Currencies representation are linked to the
local?
2. True or False: It is not possible to search over scripted fields
3. What are 3 things to be aware of when using field scripting?
!268

Kibana Management
Lesson 1
Lab - Advanced Settings

Reporting and Saved Objects
Kibana Management
Lesson 2

Generating Report
• Kibana can generate reports on click of a button
• Reports can be generated using:
‒ Discover Interface
‒ to generate a CSV of documents in search results
‒ Results are limited to defined columns
‒ Dashboard interface
‒ to generate a pdf of the dashboard
• After report is generated, it can be downloaded from
Management -> Reporting
!271

Reporting Automation
• When it comes to reporting, automation is always
appreciable
• Using watcher it is possible to generate report
automatically:
‒ be careful, the interval of the generation of reports need to be
higher than the time it takes for a report to be generated
• Watcher can be technical, but using the following example
make it easier to set it up:
‒ https://www.elastic.co/guide/en/kibana/current/automating-report-
generation.html
!272

Saved Objects
• In Kibana, the following are referred to as objects
‒ Index pattern
‒ Visualization
‒ Dashboard
‒ Saved search
• Saved Objects
‒ Can be exported as JSON
‒ Can be Imported into another Kibana Instance
‒ Are internally stored in an Elasticsearch index
!273

Saved Objects Relationship
• Saved objects might be related, for instance
‒ Dashboards contains visualizations
‒ Visualizations may be based off of Saved Searches
‒ Saved Searches target Index patterns
• For any saved object, you can see related saved objects
‒ Will help understand implications of deleting a saved object
!274

Instructor Demo
Demo
!275

Review - Reporting and Saved
Objects
Kibana Management
Lesson 2

Summary
• Dashboards, searches and canvas can be exported for
reporting
• Dashboards can be downloaded as a PDF and searches
can be downloaded as a CSV
• Saved objects are stored inside an index in Elasticsearch
• Using Elastic Alerting (Watcher) it is possible to automate
report generation
!277

Quiz
1. True or False: Saved objects are stored inside Kibana
2. True or False: Alerting can be used to automate report
generation
3. True or False: Kibana can gives you the ability to view
related objects of another saved object
!278

Kibana Management
Lesson 2
Lab - Reporting and Saved Objects

Security and Spaces
Kibana Management
Lesson 3

Kibana_Data_analyst_7.1.0.pdf

Recommended

Recommended

More Related Content

Similar to Kibana_Data_analyst_7.1.0.pdf

Similar to Kibana_Data_analyst_7.1.0.pdf (20)

Recently uploaded

Recently uploaded (20)

Kibana_Data_analyst_7.1.0.pdf