[HCM Scrum Breakfast - January 2018] ElasticSearch In Action

www.axon.vnfb.com/AxonActiveVietNam
Who we are?
Lương Văn Thắng

Who we are?
Vũ Hiển Khang

Who we are?
Đỗ Việt Trung

Theme: Learning mindset

1. OVERVIEW
2. Searching
3. SCALE
4. SECURITY
5. MIGRATION
6. LIMITATION
7. DEMO
Agenda

OVERVIEW

What is Elasticsearch?
❖ Search engine based on Lucene
❖ Real-time distributed, full-text search engine
❖ RESTful API
❖ Schema-free
❖ First public release in Feb 2010
Wikipedia

WHO are using ELASTICSEARCH?

WhY Elasticsearch?
● NoSQL DB for indexing JSON contents
● Schema-free
● Distributed
● High performance
● REST semantics
● Graph capabilities
● Great documentation
● Open source!

WhY Elasticsearch?

WORK WITH Big data

Searching – poor performance

Schema changes frequently

Scaling database

Searching – faster and smarter

Our solution

Searching In Action

Get started

❖ Node – a started instance of Elasticsearch
❖ Cluster - collection of connected nodes of Elasticsearch
Cluster and Nodes

❖ A collection of types
❖ Similar to database
Indexes

❖ Collection of documents
❖ Has schema (implicit or explicit)
❖ Similar to table
Types

❖ Self-contained data
❖ Have id
❖ Have schema
❖ Similar to record
Documents

❖ Documents are structured in fields
❖ Special fields: _id, _uid, _index, _type, _all, _source, …
❖ Similar to column
Fields

❖ text, keyword
❖ long, integer, short, double, float, byte
❖ date
❖ boolean
❖ binary
❖ geo
❖ object
❖ nested
❖ ip
❖ arrays
❖ …
Data types

Mappings and settings
Mappings – schema of documents
+ dynamic mappings
+ explicit mappings
Settings – configured for each index
+ good enough when using default settings
+ index analysis – expert users

Download and setup
https://www.elastic.co/guide/en/elasticsearch/reference/5.4/setup.html

Setup Elasticsearch

{
"from": 0,
"size": 20,
"min_score": 5,
"query": {
"bool": {
"should": [],
"must": [],
"must_not": [],
"filter": {}
}
},
"sort": [],
"aggs": {}
}
Query structure

{
"from": 0,
"size": 20,
"min_score": 5,
"query": {
"bool": {
"should": [],
"must": [],
"must_not":
[],
"filter": {}
}
},
"sort": [],
"aggs": {}
}
Query structure
// do pagination

{
"from": 0,
"size": 20,
"min_score": 5,
"query": {
"bool": {
"should": [],
"must": [],
"must_not": [],
"filter": {}
}
},
"sort": [],
"aggs": {}
}
Query structure
// limit hitlist return

{
"from": 0,
"size": 20,
"min_score": 5,
"query": {
"bool": {
"should": [],
"must": [],
"must_not": [],
"filter": {}
}
},
"sort": [],
"aggs": {}
}
Query structure
// statistic

{
"from": 0,
"size": 20,
"min_score": 5,
"query": {
"bool": {
"should": [{..}, {..},..],
"must": [{..}, {..},..],
"must_not": [{..}, {..},..],
"filter": {}
}
},
"sort": [],
"aggs": {}
}
Query structure
// and
// or
// not

{
"bool": {
"must": [
{“strong”},
{“is man”}
]
}
}
Exercise 1

{
"bool": {
"must": [
{“strong”},
{“is man”}]
}
}
Exercise 1

{
"bool": {
"must":
[{“has red on suite”}],
“should“:
[{“strong”},{“is man”}]
}
}
Exercise 2

{
"bool": {
"must": [{“has red on suite”}],
"should":[{“strong”},{“is man”}]
}
}
Exercise 2

Exercise 3
{
"bool": {
"must":
[{“strong”},{“has
cape”}],
“must_not“:
[{“be seen the face”} ]
}
}

{
"bool": {
"must":
[{“strong”},
{“has cape”}],
“must_not“:
[{“be seen the face”}]
}
}
Exercise 3

❖ Mail-search server
❖ Dataset of >470,000 emails
❖ Legal copy from a dissolved
company - US
Workflow of building a query

❖ Has ‘bomb’ in content
❖ Sent by steven.kean@enron.com
❖ Interested in receiver
bill.donovan@enron.com

Synonym

Synonym token filter

SCALE

What – Why?

How to Scale

SECURITY

WHY
Elasticsearch has no concept of a user.
Essentially, anyone that can send arbitrary requests to your
cluster is a “super user”.

WHAT
❖ Authentication & authorization
❖ Access control
❖ Encryption
❖ Auditing

Authentication
& Authorization
Role-based
access control
Encryption
Auditing
License PaidPaid/UnpaidPaid/Unpaid Paid/Unpaid

Prevent unauthorized access
Keep Data Integrity
what happen on system

Installing….

User Authentication

Identify who send the request

built-in users
elastic kibana logstash_system

ADD More User
PUT localhost:9200/_xpack/security/user/trungdo
{
"full_name" : "Trung Do",
"email" : "trung.do@axonactive.com",
"password" : "axonvn",
"roles" : [ "admin", "superior", "kibana_user" ],
"metadata" : {
"workingYear" : 5
},
"enabled": true
}

How authentication works
Native
(Basic Auth)
LDAP
file
(Basic Auth)
Active
Directory
PKI Custom
Realms

Realms chain
Native file LDAP1 LDAP2

xpack.security.authc:
realms:
file: //id of realm
type: file //type of realm
order: 0 //order in chain
native:
type: native
order: 1
ldap1:
type: ldap
order: 2
enabled: false
url: 'url_to_ldap1'
…
ldap2:
type: ldap
order: 3
enabled: false
url: 'url_to_ldap2'
Configure realms
chain
elasticsearch.yml

Authorization

❖ Identify permission to execute request
❖ Support by Role Based Access Control (RBAC)

Secured Resource
RBAC
Role
Privilege
Permissions
User

Secured Resource
Object need to be restricted accessity
Indices Document Field
User Cluster

Privilege
❖ One or group of actions user can execute
❖ Two types: Cluster and Indice
all read create index delete
See full privileges: https://www.elastic.co/guide/en/x-pack/5.4/security-privileges.html
all monitor manage_watcher manage_security
Indice privilege
Cluster privilege

Permission
Set of privileges on a secured resource
read on kibana
indices
monitor on cluster
write on kibana
indices
kibana system permission

Built-in Role
superuser kibana_user kibana_system logstash_system
monitoring_user reporting_user watcher_admin watcher_user machine_learning_user
See more at: https://www.elastic.co/guide/en/x-pack/5.4/built-in-roles.html

ADD ROLE
POST localhost:9200/_xpack/security/role/scrum_admin
{
"run_as": [ ... ],
"cluster": [ ... ],
"indices": [ ... ]
}
//submit request on behalf of other user
//privilege on cluster
//permission entry on indices

ADD ROLE
POST localhost:9200/_xpack/security/role/scrum_admin
{
"run_as": [ "khangvu" ],
"cluster": [ "monitor" ],
"indices": [
{
"names": [ "scrum*" ],
"privileges": [ "read" ],
"field_security" : {
"grant" : [ "category", "@timestamp", "message" ]
},
"query": "{"match": {"event.type": "technical"}}"
}
]
//scrum_admin can submit request for khangvu user
//can monitor cluster
//name of targeted-indices: all indexes started with scrum
//privilege on targeted-indices
//allowed field to access
//list of targeted-document

Auditing
❖ Audit activities/events that occur in the system
❖ Output to logfile or index
❖ Enable this feature by
xpack.security.audit.enabled to true in elasticsearch.yml.
See more at https://www.elastic.co/guide/en/x-pack/5.4/auditing.html

Encrypting communication
❖ Encrypt traffic to, from and within an ES cluster using
SSL/TLS certificate.
❖ Able to increase strength of encryption by Java
Cryptography Extension (JCE) plugin
❖ Separate port of node-to-node and transport client
See more at https://www.elastic.co/guide/en/x-pack/5.4/encrypting-communications.html

migration with logstash

missions
❖ Data migration tool
❖ Lightweight
❖ Scalabled
❖ Managabled
❖ Monitoring UI

Logstash is data collector pipeline

How ?
❖ Event processing pipeline
❖ Has three stages:
➢ Input
➢ Filter
➢ Output
❖ Has bunch of plugins for you to play with

usecase - Collect rdbms to es server
Database Logstash filter ElasticSearch
Store Read
Kibana

Database Logstash ElasticSearch
StoreRead

ElasticSearch
Read
Kibana

Input stage
❖ Plugins:
➢ jdbc
➢ file
➢ kafka
➢ redis
➢ beats
➢ …
See more at https://www.elastic.co/guide/en/logstash/current/input-plugins.html

Filter stage
❖ Plugins:
➢ grok
➢ mutage
➢ geoip
➢ ...
See more at https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

output stage
❖ Plugins:
➢ elasticsearch
➢ file
➢ redis
➢ kafka
➢ ...
See more at https://www.elastic.co/guide/en/logstash/current/output-plugins.html

scaling
Server-01
Logstash filter ElasticSearch
Read
Kibana
shipper-1
Garther Store
shipper-2
shipper-3
Server-02
Server-03
your business, Your imagination

LIMITATION

❖ ES has no transaction management, can’t rollback data.
❖ Problem with index when change data property
{
"firstname": "Trung" ,
"lastname": "Do",
"email": "trung.do@axonactive.com" ,
"phone": "0909999999" ,
"street": "39B Truong Son" ,
"city": "Ho Chi Minh"
}
{
"firstname": "Trung" ,
"lastname": "Do",
"contact": {
"email": "trung.do@axonactive.com" ,
"phone": "0909999999" ,
"street": "39B Truong Son" ,
"city": "Ho Chi Minh"
}
}

[HCM Scrum Breakfast - January 2018] ElasticSearch In Action

Recommended

Recommended

More Related Content

Similar to [HCM Scrum Breakfast - January 2018] ElasticSearch In Action

Similar to [HCM Scrum Breakfast - January 2018] ElasticSearch In Action (20)

More from Scrum Breakfast Vietnam

More from Scrum Breakfast Vietnam (20)

Recently uploaded

Recently uploaded (20)

[HCM Scrum Breakfast - January 2018] ElasticSearch In Action